Six essential skills for Big data

Analytical Skills

Analytics involves the ability to determine which data is relevant to the question that you are hoping to answer, and interpreting the data in order to derive those answers.

If you have a knack for spotting patterns, and establishing links between cause and effect, then these skills will prove invaluable if you’re tasked with turning a business’s data into actionable plans of operation.

Creativity

There are no hard and fast rules about what a company should use big data for. It is an emerging science, which means the ability to come up with new methods of gathering, interpreting, analysing and – finally – profiting from – a data strategy, is a very valuable skill.

The corporate data superstars of the future will be people who can come up with new methods of applying data analytics in innovative ways. Often they will be solving problems that companies don’t even know they have – as their insights highlight bottlenecks or inefficiencies in the production, marketing or delivery processes. In particular, creativity is important for anyone hoping to make sense of unstructured data – data which does not fit comfortably into tables and charts, such as human speech and writing.

Mathematics and Statistics

Good old fashioned number crunching. Despite the growing amount of unstructured data being incorporated into data strategies, much of the information being gathered and stored, ready for analysis, still takes the form of numbers.

And even when dealing exclusively with unstructured data, the objective of the exercise is often to reduce elements of the data – emails, social media messages etc – to figures which can be quantified, in order for definite conclusions to be drawn from them. This means candidates with a strong background in maths or statistics are ideally placed to make the leap into big data enterprise.

Computer Science

Computers are the workhorses behind every big data strategy, and programmers will always be needed to come up with the algorithms that process data into insights. This is a very broad category which covers a whole range of subfields, such as machine learning, databases or cloud computing, which will be great additions to any budding data scientist’s arsenal. In particular you should be familiar with the range of open-source technologies – Hadoop, Python, Pig etc. – which make up the foundations of most big data enterprises.

Business skills

An understanding of business objectives, and the underlying processes which drive profit and business growth are also essential. The idea that a company will hire an “egg head” data scientist who will be locked away in a basement lab, to work their magic on data fed to them through a slot in their door, is dangerous and wrong. They should have a firm grasp of the company’s business goals and objectives as well as an understanding of the indicators which let them know if they are heading in the right direction.

Communication ability

Both inter-personal and written – an essential part of a data scientist skillset is the ability to communicate the results of the analysis to other members of their team as well as to the key decision-makers who need to be able to quickly understand the key messages and insights.

This also includes the skills of visualising and reporting data in the most effective manner. You can have the best analytical skills in the world, but unless you are able to make your findings understandable to everyone else you work with, and demonstrate how they will help to improve performance and drive success, they will be of little use to any business.

-----

Smart Grid Analytics

World Market for $1 trillion is being invested this decade in upgrading the power infrastructure globally to make the devices in the power grid remotely machine addressable. These devices include meters, thermostats, home appliances and HVAC equipment, factory equipment and machinery, and transformers, substations, distribution feeders, and power generation and control componentry.

Till Now 310 million smart meters have been installed globally. That number will more than triple by 2022, reaching nearly 1.1 billion according to Navigant Research. While representing only a fraction of the sensors on the grid infrastructure, the smart meter installation numbers provide a good indication of the penetration and rate of growth of the smart grid. These developments are occurring worldwide.

Collectively, these devices generate massive amounts of information. With recent developments in information technology, including elastic cloud computing and the sciences of big data, machine learning, and emerging social human-computer interaction models, we are able to realize the economic, social, and environmental value of the smart grid by aggregating the sum of these data to correlate and scientifically analyze all of the information generated by the smart grid infrastructure in real time.

By holistically correlating and analyzing all of the dynamics and interactions associated with the end-to-end power infrastructure—including current and predicted demand, consumption, electrical vehicle load, distributed generation capacity, technical and non-technical losses, weather, and generation capacity— across the entire value chain, we can realize dramatic advances in energy efficiency.

Smart grid analytics enables us to provide real-time pricing signals to energy consumers, manage sophisticated energy efficiency and demand response programs, conserve energy use, reduce the fuel necessary to power the grid, reconfigure the power network around points of failure, recover instantly from power interruptions, accurately predict load and distributed generation capacity, rapidly recover from damage inflicted by weather events and system failures, and reduce adverse environmental impact.

The advent of smart grid analytics represents a major advance in the development of energy efficiency technology. Many leading utilities including Enel, GDF Suez, Exelon and PG&E work with us to drive innovation by applying the science of smart grid analytics to the benefit of their communities, consumers, and stakeholders.

DATA SCIENTIST

Step 1: Graduate from a top tier university in a quantitative discipline

. Education makes a huge difference in your prospects to start in this industry. Most of the companies who do fresher hiring, pick out people from best colleges directly. So, by entering into a top tier university, you give yourself a very strong chance to enter data science world.

Ideally I would take up Computer Science as the subject of study. If I didn’t get a seat in Computer Science batch, I’ll take up a subject which has close ties with computational field - e.g. computational nueroscience, Computational Fluid Dynamics etc.

Step 2: Take up a lot of MOOCs on the subject – but do them one at a time

This is probably the biggest change, which would happen in the journey, if I was passing out now. If you spend even a year studying the subject by participating in these open courses, you will be in far better shape vs. other people vying to enter the industry. It took me 5+ years of experience to relate to the power R or Python bring to the table. You can do this today by various courses running on various platforms.

One word of caution here is to be selective on the courses you choose. I would focus on learning one stack – R or Python. I would recommend Python over R today – but that is a personal choice. You can find my detailed views about how the eco-systems compare here.

You can choose your path – but this is probably what I would do:

• Python:
• Introduction to Computer Science and Programming using Python – eDX.org
• Intro to Data Science – Udacity
• Workshop videos from Pycon and SciPy – some of them are mentioned here
• Selectively pick from the vast tutorials available on the net in form of iPython notebooks
• R:
• The Analytics Edge – eDX.org
• Pick out a few courses from Data Science specialization to complement Analytics Edge
• Other courses (applicable for both the stacks):
• Machine Learning from Andrew Ng – Coursera
• Statistics course on Udacity
• Introduction to Hadoop and MapReduce on Udacity

Step 3: Take a couple of internships / freelancing jobs

This is to get some real world experience before you actually venture out. This should also provide you an understanding of the work which happens in the real world. You would get a lot of exposure to real world challenges on data collection and cleaning here. 

Step 4: Participate in data science competitions

You should aim to get at least a top 10% finish on Kaggle before you are out of your university. This should bring you in eyes of the recruiters quickly and would give you a strong launchpad. Beware, this sounds lot easier than what it actually is. It can take multiple competitions for even the smartest people to make it to the top 10% on Kaggle.

Here is an additional tip to amplify the results from your efforts – share your work on Github. You don’t know which employer might find you from your work!

Step 5: Take up the right job which provides awesome experience

I would take up a job in a start-up, which is doing awesome work in analytics / machine learning. The amount of learning you can gain for the slight risk can be amazing. There are start-ups working on deep learning, re-inforcement learning – choose the one which fits you right (taking culture into account)

If you are not the start-up kinds, join a analytics consultancy, which works on tools and problems across the spectrum. Ask for projects in different domains, work on different algorithms, try out new approaches. If you can’t find a role in a consultancy – take up a role in captive units, but seek a role change every 12 – 18 months. Again this is a general guideline – adapt it depending on the learning you are having in the role.

Finally a few bonus tips:

• Try learning new tools once you are comfortable with ones you are already using. Different tools are good for different types of problem solving. For e.g. Learning Vowpal Wabbit can add significant advantage to your Python coding.
• You can try a shot at creating a few web apps – this adds significant knowledge about data flow on the web and I personally enjoy satisfying the hacker in me at times!

Few modifications to these tips, in case you are already out of college or hold work experience:

• In case you can still go back to college, consider getting a Masters or a Ph.D. Nothing beats the improvement in probability of getting the right job compared to undergoing a good programme from top notch University.
• In case full time education is not possible, take up a part time programme from a good institute / University. But be prepared to put in extra efforts outside these certifications / programmes.
• If you are already in a job and your company has an advanced analytics setup, try to get an internal shift by demonstrating your learning.
• I have kept the focus on R or Python, because they are open source in nature. If you have resources to get access to SAS – you can also get a SAS certification for predictive modeler. Remember, SAS still holds the majority of jobs in analytics!

Big Data for Industries

4 key layers of a big data system - i.e. the different stages the data itself has to pass through on its journey from raw statistic or snippet of unstructured data (for example, social media post) to actionable insight.

Data sources layer

This is where the data is arrives at your organization. It includes everything from your sales records, customer database, feedback, social media channels, marketing list, email archives and any data gleaned from monitoring or measuring aspects of your operations. One of the first steps in setting up a data strategy is assessing what you have here, and measuring it against what you need to answer the critical questions you want help with. You might have everything you need already, or you might need to establish new sources.

Database storage layer

This is where your Big Data lives, once it is gathered from your sources. As the volume of data generated and stored by companies has started to explode, sophisticated but accessible systems and tools have been developed – such as Apache Hadoop DFS (distributed file system), – or Google File System, to help with this task. A computer with a big hard disk might be all that is needed for smaller data sets, but when you start to deal with storing (and analyzing) truly big data, a more sophisticated, distributed system is called for. As well as a system for storing data that your computer system will understand (the file system) you will need a system for organizing and categorizing it in a way that people will understand – the database. Hadoop has its own, known as HBase, but others including Amazon’s DynamoDB, MongoDB and Cassandra (used by Facebook), all based on the NoSQL architecture, are popular too. This is where you might find the Government taking an interest in your activities – depending on the sort of data you are storing, there may well be security and privacy regulations to follow.

Database  processing/ analysis layer

When you want to use the data you have stored to find out something useful, you will need to process and analyze it. A common method is by using a MapReduce tool. Essentially, this is used to select the elements of the data that you want to analyze, and putting it into a format from which insights can be gleaned. If you are a large organization which has invested in its own data analytics team, they will form a part of this layer, too. They will employ tools such as Apache PIG or HIVE to query the data, and might use automated pattern recognition tools to determine trends, as well as drawing their conclusions from manual analysis.

Database  output layer

This is how the insights gleaned through the analysis is passed on to the people who can take action to benefit from them. Clear and concise communication (particularly if your decision-makers don’t have a background in statistics) is essential, and this output can take the form of reports, charts, figures and key recommendations. Ultimately, your Big Data system’s main task is to show, at this stage of the process, how measurable improvement in at least one KPI that can be achieved by taking action based on the analysis you have carried out.

Analytics to Utility Industry

1.     Analytics can help make the grid safer. 
Utilities need to beef up security when it comes to smart grids. A breach in smart grid security could spell disaster for utilities. Analytics could be used to help detect if someone is in the grid framework, tampering with meters and other tools. They could also see energy being improperly diverted. It’s already begun to happen and will continue. Smart grids are a powerful too, but they’re also highly vulnerable. Analytics could help maintain and beef up security

2. drive innovation and ultimately job growth. 
IT professionals are always in demand, but especially when it comes to the development side of the smart grid.
“Market for IT solutions will be one of--if not the--largest categories among all of the components of a smart grid. “The only larger category is transmission upgrades, which will be necessary for the grids of the future, but aren't 100% tied to actual 'smart' technology.”

3. Better understanding how the grid works allows for better efficiency.
Analytics help understand exactly how the grid works and is able to identify trends. Using this knowledge, utilities can help drive efficiency and better utilize their resources. This is something that must be done, too.
“Growth in smart grid is no longer a luxury for utilities—aging infrastructure, retiring personnel and the proliferation of distributed generation resourceS

4.Forecast severe weather events and other emergencies that could disrupt the grid.
One of the biggest issues with aging grids is the fact that anything can take them down at any given time. With analytics, these events can be better predicted and prepared for. Severe storms are more and more common, making grid outages more common.
“Analytics enable operators to monitor and report the exact times of service interruption at each system endpoint and use these results to measure improvement in restoration time from automated distribution processes,” . “This allows utilities to identify and restore outages more rapidly, without having to rely on customer inquiries.”
This means faster response times and faster repairs.

5. Manage the load balance.
As grids age, properly balancing the power distribution can sometimes be a bit tricky. However, by using smart grid analytics, power can be distributed as needed in an effective manner. .
Transformer load management analytics can utilize smart-meter data and actual  to continuously monitor and analyze distribution transformer loading levels and report on asset health, helping utilities make informed decisions to balance loads,”

Smart Grid – Key to Managing Energy Demand in India

Electricity consumption in  india has been climbing steadily for the past few decades. THE electricity market is growing at an accelerating rate due to higher consumption rates in the private, commercial and industrial sectors. Current domestic energy consuming behaviors pose unescapable fatal consequences that affect both  production and export levels. Therefore, an urgent action is needed to curb the increasing electricity demand and promote energy conservation. Smart grid is a dynamic solution which can bridge the gap between the current supply and increasing demand in india

What is smart grid?

A smart grid network makes for the ideal bridge where the goals of modernization can meet those of a reliable public infrastructure. Smart grid is a computerized technology, based on remote control network, aiming to completely alter the existing electric infrastructure and modernize the national power grid. This is through empowering the demand response which alerts consumers to reduce energy use at peak times. Moreover, demand response prevents blackouts, increases energy efficiency measures and contributes to resource conservation and help consumers to save money on their energy bills. Smart grid technology represents an advanced system enabling two way communications between energy provider and end users to reduce cost save energy and increase efficiency and reliability.

Advantages of smart grid

The beauty of adapting this technology will spread to not only utility but to all utility users including consumers and government.

Active role of consumer.

The beauty of smart grid is that it provides consumers with the ability to play an active role in the country’s electricity grid. This is through a regulated price system where the electricity rate differs according to peak hours and consequently consumers cut down their energy use at those high stress times on the grid. Thus, smart grid offers consumers more choices over their energy use needs. 

Upgrading the Existing Grid

Utilities benefit from improving the grid’s power quality and reliability as mentioned through an integrated communication system with end users with more control over energy use. This is through decreasing services rates and eliminating any unnecessary energy loss in the network. Thus, all these positive advantages will make smart grid technology a smart and efficient tool for utilities.

Contributing to Energy Efficiency

The government of India is in the process of taking bold steps to adapt new energy efficiency standards to reduce domestic energy consumption. For that, adapting and deploying smart grid will enable to modernize the national grid. With the time the government will build efficient and informed consumers as a backbone in its current energy policy. Moreover, this advanced technology will help with electricity reduction targets and contribute to lowering the carbon dioxide emissions. Thus, this is a great opportunity for the kingdom to mitigate with the climate change measures.

A Dynamic Approach

Adoption of smart grid systems will help India in increasing the efficiency of utilities as well as improving the ability of consumers to control their daily energy use. Smart grid technology offers a unique engagement that benefits consumers, utilities and government to become part of the solution. In addition, a smart grid technology is a viable option to enhance the value people receive from the national grid system. This smart transition will give the government a policy option to reduce drastically its domestic energy use, leveraging new technology through empowering the role of consumers’ active participants on the country’s grid.

As peak electricity demand grows across the country, it is important for India to make large-scale investment in smart grid solutions to improve energy efficiency and manage increasing energy demand. Undoubtedly, smart grid is more intelligent, versatile, decentralized, secure, resilient and controllable than conventional grid. However, to reap the benefits of smart grid systems, utilities  need to make major changes in their infrastructure and revolutionize the manner in which business is conducted.

Off grid Photovoltaic shelter for streetlightning

Today, some part of the Indians have no access to electricity. During night the  life comes to a halt, affecting  productivity, employment and quality of life.

Off-grid LED street lighting solution has been designed for areas with limited or no access to electricity and use a completely sustainable system.

When the sun shines during the day, the solar panels convert solar energy to electrical energy and stores it in the solar-powered battery. During the night, the battery is discharged, releasing electrical energy to power the LED lamps and lighting hence the road.

The primary advantage of LED street lighting is energy efficiency; in fact the LED technology ensure low power consumption, long lifetime, more accurate color rendering and fewer electrical losses.

Furthermore, by installing efficient and bright street lighting the quality of life get better; the local authorities can improve road safety for drivers and pedestrians, create more attractive conditions for businesses and commercial life  and generates  a  more secure life