5 Big Data Technology Predictions for 2015

1. Data Agility Emerges as a Top Focus

Data agility has been one of the big drivers behind the development of big data technologies, as the processes around legacy databases and data warehouses have proven too slow and inflexible for many business needs. In 2015, Schroeder says data agility will become even more central as organization shift their focus from simply capturing and managing data to actively using it.
"Legacy databases and date warehouses are so expensive that DBA resources are required to flatten summarize and fully structure the data," he says. "Upfront DBA costs delay access to new data sources and the rigid structure is very difficult to alter over time. The net result is that legacy databases are not agile enough to meet the needs of most organizations today."
[Related: 8 Big Trends in Big Data Analytics]
"Initial big data projects focused on the storage of target data sources," he adds. "Rather than focus on how much data is being managed, organizations will move their focus to measuring data agility. How does the ability to process and analyze data impact operations? How quickly can they adjust and respond to changes in customer preferences, market conditions, competitive actions and the status of operations? These questions will direct the investment and scope of big data projects in 2015."

2. Organizations Move from Data Lakes to Processing Data Platforms

Thinkstock

Featured Resource

Presented by Citrix Systems

10 Essential Elements for a Secure Enterprise Mobility Strategy

Best practices for protecting sensitive business information while making people productive from
Learn More

In some ways, 2014 was the year of the data lake (or data hub), an object-based storage repository that stores raw data in its native format — whether structured, unstructured or semi-structured — until it's ready for use. Data lakes have a strong value proposition in that they represent a scalable infrastructure that's economically attractive (with a reduced per-terabyte cost) and extremely agile.
Schroeder says that the data lake will continue to evolve in 2015 with the capability to bring multiple compute and execution engines to the data lake to process the data in-place. That's not only more efficient, it creates a single point of governance and a single point of security.

 

 

[Related: C-Level Executives Seeing Big Results From Big Data ]
"In 2015, data lakes will evolve as organizations move from batch to real-time processing and integrate file-based, Hadoop and database engines into their large-scale processing platforms," he says. "In other words, it's not about large-scale storage in a data lake to support bigger queries and reports; the big trend in 2015 will be around the continuous access and processing of events and data in real time to gain constant awareness and take immediate action."

3. Self-Service Big Data Goes Mainstream

Thinkstock

Advances in big data tools and services means that 2015 will be the year that IT can ease away from being a bottleneck to the access of data by business users and data scientists, Schroeder says.
"In 2015, IT will embrace self-service big data to allow business users self-service to big data," he says. "Self-service empowers developers, data scientists and data analysts to conduct data exploration directly."

 

 

Previously, IT would be required to establish centralized data structures," he adds. "This is a time-consuming and expensive step. Hadoop has made the enterprise comfortable with structure-on-read for some use cases. Advanced organizations will move to data bindings on execution and away from a central structure to fulfill ongoing requirements. This self-service speeds organizations in their ability to leverage new data sources and respond to opportunities and threats."

4. Hadoop Vendor Consolidation: New Business Models Evolve

Thinkstock

Featured Resource

Presented by Citrix Systems

10 Essential Elements for a Secure Enterprise Mobility Strategy

Best practices for protecting sensitive business information while making people productive from
Learn More

In early 2013, Intel made a splash with the introduction of its own Hadoop distribution, saying that it would differentiate itself by taking a ground-up approach in which Hadoop was baked directly into its silicon. But just a year later, Intel ditched its distribution and threw its weight behind Hadoop distribution vendor Cloudera instead.
At the time, Intel noted that customers were sitting on the sidelines to see how the Hadoop market would shake out. The number of Hadoop options were muddying the waters. Schroeder believes Hadoop vendor consolidation will continue in 2015 as the also-rans discontinue their distributions and focus elsewhere in the stack.

 

 

[Related: The CIO and CMO Perspective on Big Data]
"We're now 20 years into open source software (OSS) adoption that has provided tremendous value to the market," Schroeder says. "Technologies mature in phases. The technology lifecycle begins with innovation and the creation of highly differentiated products and ends when products are eventually commoditized. [Edgar F.] Codd created the relational database concept in 1969 with innovation leading to the Oracle IPO in 1986 and commoditization beginning with the first MySQL release in 1995. So historically, database platform technology maturity took 26 years of innovation prior to seeing any commoditization."
"Hadoop is early in the technology maturity lifecycle with only 10 years passing since the seminal MapReduce white papers were published by Google," he adds. "Hadoop adoption globally and at scale is far beyond any other data platform just 10 years after initial concept. Hadoop is in the innovation phase, so vendors mistakenly adopting "Red Hat for Hadoop" strategies are already exiting the market, most notably Intel and soon EMC Pivotal."
Schroeder believes 2015 will see the evolution of a new, more nuanced model of OSS that combines deep innovation with community development.
"The open source community is paramount for establishing standards and consensus," he says. "Competition is the accelerant transforming Hadoop from what started as a batch analytics processor to a full-featured data platform."

Thinkstock

Featured Resource

Presented by Citrix Systems

10 Essential Elements for a Secure Enterprise Mobility Strategy

Best practices for protecting sensitive business information while making people productive from
Learn More

5. Enterprise Architects Separate the Big Hype from Big Data

2015 will see enterprise architects take center stage as their improving understanding of the Hadoop technology stack leads to a better defined and more sophisticated statement of requirements for big data applications, including elements like high availability and business continuity.
"As organizations move quickly beyond experimentation to serious adoption in the data center, enterprise architects move front and center into the big data adoption path," Schroeder says. "IT leaders will be vital in determining the underlying architectures required to meet SLAs, deliver high availability, business continuity and meet mission-critical needs. In 2014 the booming ecosystem around Hadoop was celebrated with a proliferation of applications, tools and components. In 2015 the market will concentrate on the differences across platforms and the architecture required to integrate Hadoop into the data center and deliver business results.

10 Characteristics Of A Bad Software Engineer

These are some characteristics of bad software engineers:

1) The StackOverflow bot: This person ran into an error, did a quick Google search, and applied the first solution they found. The problem here is not that of copying from Stackoverflow. I think there are more solutions on Stackoverflow than any reference guide or manual. Don’t get me wrong, it’s a wonderful resource, if not the best. The problem is the robotic application of it without understanding the consequences. The problem is the application of it without fully understanding the context of it and whether it really applies to the current problem at hand. More often than not, I have seen people believe more of what they see on online forums than the code/system in front of them.

2) The I-am-not-a-tester: I don’t need to test the code, that is the job of the testers. I don’t think that even in this age of mature Agile methodologies, this attitude has waned. There is still an inertia against testing their code. Part of it comes from lacking the interest to set up a testing environment and partly from lack of coherent knowledge of testing. (Is it also partly due to an unspoken stigma against testers in the developer community.)

3) The I-hate-documentation: Some people believe that code documentation must be poetic and hence they lack the skill to do it, ergo not their job. In my opinion, these are the #1 foes of sustainable software. Good software is not software that provides a million cool features. Good software is one that has a few good features that are used consistently by many people and read/updated/modified by a thousand. This brand of developers who believes less in technical communication and precise and detailed documentation is the greatest weed to a company’s success.

4) The ugly: My code works, but:

-I have variables named x, flag, str, arr, etc.

-Most of what I write is in one giant method.

-There is no indentation.

-No consistent coding convention or style.

-Global variables spewed all over the place, etc.

This is the most annoying thing for me personally. It’s not the issue that the code is bad. It could potentially be the greatest piece of code written. But if a diamond necklace is buried in the debris of the Titanic, nobody will find it, and nobody will want to clean it, wear it, use it.

5) The short-term investor: He codes. He deploys. He moves on. No attempt to learn the problem. No interest in the domain. Just give this guy a piece of code, he will slog on it overnight and hand it over. You got a fix/working software. Nothing more achieved from it. Sometimes, it’s important that you have certain selfishness in the developer, one who not only cares about the deadline, but also cares about what he/she got to learn from it.

6) The protester: “I didn’t do this”. “This looks bad”. “Not my problem”. “This isn’t related really to my fix, but someone way over there made a mistake”. “I hate this (loop this sentence 10 times a day)”, “I can’t fix this, get the person who made this code to fix it”.

The person who coded that mistake has moved on, when will you?

7) The dictator: My way or the highway is their motto. It’s their “ideas” vs “your ideas”, not “project ideas”. It’s their solution vs your solution. I bet there will be an argument for sure. Somehow they will keep coming back to a part of code that you implemented. It somehow discomforts them even if it works, tests, and looks perfectly fine. This person is a big bottleneck to productivity and will be the first person to crumble under pressure and start pointing fingers. This person is not good for the team, however experienced/good a developer he may be.

8) The overcautious: The Java developer who just froze when he learned that he would have to write a Python script. The developer who panicked on learning that something in the registry needs changing. The developer who cringes at having to input things in the database. These people will do anything to avoid getting out of their comfort zone. They have weird superstitions related to having to touch certain parts of the system. I have learned, from personal experience, that this phenomenon is common with new developers. Good developers show a tendency to slowly/swiftly move out of their comfort zone in exploration.

9) The careless: Forgets to take a backup, snapshots, has multiple working directories of code, leaves system out, prints in production code, etc. Again, this is a newbie tendency and gets better with more professional exposure.

10) The lazy pseudo-hacker: They pride themselves at being able to trick the system into working. They find magical solutions to seemingly complex problems. My experience says that 9 out of 10 times, it’s just a facade. The hack is bad and will crash sooner or later and will cost much more than having to deal with it, with extra time right now.

Posted By: anuj chchabra