Below, as we explore the dataset together, I will show you how to create some basic plots and look into datasets. The Dataset is called “Yearly electricity data” and it has been put together by ember. You can find it here. Dataset description reads:

This dataset contains yearly electricity generation, capacity, emissions, import and demand data for over 200 geographies. Data is collected from multi-country datasets (EIA, Eurostat, BP, UN) as well as national sources (e.g China data from the National Bureau of Statistics).

I am using Python in an interactive mode (Jupyter in VSCode) for this tutorial. If you want to learn how to use Python before jumping into data visualisation, please read my first two articles here and here.

I start with Downloading the set, loading the main two libraries and reading the data into a Pandas DataFrame:

import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_csv('yearly_full_release_long_format.csv')

a simple df.head() will help us to read the first 5 rows of the data and see what has been presented to us. This also help to see if the data is read the way we were expecting it:

This shows us the data consist of Area and Country code followed by the year that data was collected. Area can be country or continent. We also have columns for regions and various groups that the country can be member of.

If we pause here for a second, you can appreciate how powerful these metadatas are and how these handful of columns can help us to report the values on many different levels. Category, Subcategory, Variable, Unit and Value are all the available electricity generation, capacity, emissions, import and demand data. The last two columns are the change calculations and we are not going to use theme here.

Let’s start with some basic exploring. For example, I am interested to know what categories are listed under Subcategory column. A simple way to find this information is to find the unique values in the column, as below:

df['Subcategory'].unique()

>>> array(['Aggregate fuel', 'Fuel', 'Demand', 'Demand per capita', 'Total', 'Electricity imports', 'CO2 intensity'], dtype=object)

Another exploration method is if I want to filter my Subcategory to only shows Demand per capita. I can simple do it as below:

Exploring the data for specific countries are very simple. Just like above, I can see the data related to Australia as below:

In the next step, I am going to explore the Demand per capita and the total demand for each country further. Once I start showing the Demand per capita, it would be obvious why we need both these views to get a better understanding of the data. Since it is an exploratory work, I am going to only create a view to the data for plotting purposes. The two views are created as below:

df_demand_per_capita_MWh = df[(df['Subcategory']=='Demand per capita') & (df['Area type']=='Country')][['Year', 'Area', 'Value']].reset_index(drop=True)

df_demand_TWh = df[(df['Subcategory']=='Demand') & (df['Area type']=='Country')][['Year', 'Area', 'Value']].reset_index(drop=True)

Here using two conditions I am selecting Demand per capita (or Demand), while only selecting the Country values. From the resulting table I am selecting the Year, Area and the Value. I am also asking Pandas to reset the indexing to make it easier to wrangle this new set.

We can go ahead and check the head of the data again to make sure the outcome is as expected. Here I use a combination of head and tail to explore rows further down the table:

As you can see the name of my DataFrame carries the name of the table and the unit system. I prefer a naming like this The benefits of this naming technique is I can remove columns of units and name from the dataset, making it lighter and easier to wrangle with.

Now looking at this table we are having a long table which is not the easiest to slice for TimeSeries data. It would be much easier if each Country was a column by itself, with year being the index and values (demand per capita MWh) as the cell values. Pandas have an easy to use Pivot function to help us with that:

Now that we have the data in the shape we are happy with, we can start the plotting journey. Python has plenty of libraries for Visualisation. For this tutorial I chose Matplotlib as it is generally the one that everyone starts on.

For the demand, let’s start with plotting the World’s total demand. For this I sum the demand on each row, which would be the total demand for all the countries for each year. The code and the chart would look like this:

If my x axis is Year or Data, I generally prefer to not label is as it is obvious to the reader and only take valuable space from my plot area.

A quick look to the plot and the followings stands out to me:

• The overall trend

• The dip in electricity demand during COVID and GFC

• The dip at 2022.

The overall growing trend, screams how the demand is soaring.

Covid and GFC dips are very interesting to me. This just shows how Electricity and by extension, energy, is so fundamental to our lives. Any changes in any aspect of our lives, shows itself in our Energy (Electricity) consumption.

For the dip at 2022, further exploration shows, the dip is not real. There is a lot of missing data in the year 2022, hence the same dataset next year should give us a more realistic picture for 2022.

So while simple plots can reveal interesting information, we should be careful to not read too much into any of the plots. We will see another example of this when we explore demand per capita.

Now that we saw the global demand, It would be interesting to see what countries have the highest demands. To make it easier to make conclusions, I sort the dataset based on the countries with highest demand and plot their demand. In this example, I am not assigning the same Y-axis range for all the countries, so that we can see their trends over the years. If I would have gave all the countries the same range based on the highest demand, the scale made it very difficult to understand any trend in most of countries out of the first handful.

This code yields 215 plots, one for each country. I will only include the first 20 as an example:

What we see here is, China is not only the top spot but also is going through a massive increase in terms of demand. While USA ,at 2nd spot, has a flat demand over the reported period, India, at the 3rd spot, is following Chinese trend. We can see Brazil(8th) and South Korea(10th) are another top 10 countries with increasing demand.

Chine and India however have huge populations, and this demand make sense if we are talking about >25% of earth’s population. So what if we plot the same chart but this time adjust it for population. Here is how the top 20 look like for demand per capita:

At least for me the first time I look at this, the results was very interesting and surprising. However, after a bit of think about it it started to make sense.

In this charts, Iceland’s demand is very high and it is twice as big as the next country. Before ruling it out as an outlier that needs elimination, I did a quick search to confirm the validity of the data. The source of this dataset is from Eurostat based on the methodology document. This article explains these numbers a bit more in detail.

Another interesting trend is the fact that many of the top 20 countries, are those on harsher climates (extreme cold or hot) without huge populations but heavy reliance on Air conditioning. Also, interestingly China in this list is in 77th spot and India is further down at 143th spot, below countries like Palestine(139), Cuba(125) and Argentina(85).

As you can see we just started to scratch the surface with this dataset. It is has already been a long tutorial and I may come back to it in another post.

In summary, in this tutorial we saw how to import a CSV dataset, do initial familiarisation with the data, do some basic arithmetic on the data and create some basic plots. If you spent more time with this dataset and have other insights in it, please for share in the comments, or simple reply with an email. Also, if you wish the Notebook carrying all this code, please message me and I will share them with you.

Having resources to know how to learn a new skill is a critical first step. It gives you the road map, but as having a treasure map is different to having the treasure in your hands, an excellent road map is not the same as knowing enough Python to apply in your day-to-day work.

Learning Python is easy, there are endless number of resources. Similar to learning a language like French or Farsi, just learning the grammar and vocabulary may seem motivating but eventually you need to be able to practice it and finding the right project can be difficult. The project must be challenging for you but at the same time should not be too challenging that demotivate you. Over the years I develop my own methods on how to go around this challenge and I am sharing those with you today.

Below I talk about two things, firstly how to find doable projects easily to keep you motivated in your journey and secondly how to measure your progress so you never feel that the time you are spending learning is wasted and can push yourself forward.

How to find doable projects easily

This is easier said than done and I am happy to acknowledge that. I can share some scenario which worked out for me and hopefully one or some of them will work out for you as well. The list below is from ideal scenario to the most practical one based on my experience. I used a combination of third and eventually second method as I was not fortunate enough to have a work mentor with Python background. Here is the list:

1. Learning at work with a mentor

   You may be incredibly lucky, and you are learning Python on the job to work on a specific project with a mentor/coach assigning you a task and can guide you to achieve it.

   This is the best way in learning Python but needs a lot of luck as you need to be in the right place in the right time and a right mentor helping you to build your skills. If you are lucky enough to be in this situation, jump on this opportunity and do not look back and move to the next section so you learn how to keep yourself motivated.

   I was not lucky enough to be in this situation, but I have been mentoring younger engineers on delivering projects using Python and I was envying their situation.

2. Learning at work without a mentor

   This is also a great way to learn Python and apply learning. Sometimes you know something can be done in Python as you have seen it in other workplaces, on the internet, or in a conference.

   Your task here is a simple. Break the complicated task to smaller and doable chunks. Now, apply your Python learning to deliver each chunk.

   My strategy here is to not sure what I am working on with my manager till I have something clear value to business to show. The benefit of this is you do not work towards a hard deadline, and you can spend more time learning and exploring your solution without complexity of giving time estimates, resource requirements, etc.

3. Learning using personal projects

   You want to find a pain point in your daily life and use Python to solve it.

   1. You may be passionate about investing or trading and want to write a small script that suggest a buy or sell based on moving average closed prices of certain stocks.

   2. You may enjoy playing around with a Raspberry Pi. (Pi website)

   3. you may have a solar panel that you can read its daily production. (example)

   4. You may want to write a small script that calculates how much you and your partner spent each month and track your shares.

   These are some of everyday script I wrote to practice various python techniques. You can also join different communities (like pi community) and see what type of projects people have worked on and get inspired. If you have done other easy and interesting projects, please comment below.

How to measure your progress

Many start the year with a weight loss resolution. Imagine you are going to the gym and eat healthy to lose weight and meet your resolution. Your motivation comes from seeing your weight going down the next time you step on a scale or when an old pants now fits you well. Same applies in learning Python and other skills.

The trick here is to find the same quantifying progress check in Python. I found two ways so far to measure this. The easiest is to review my old codes and see how much I can improve them while the second one is how I can teach a new learner the ropes.

• For code review path, depending on the pace you are learning and the time you are investing, I suggest reviewing your old codes every few weeks or every few months. It is quite common to not appreciate how much you have learnt and hence getting demotivated as you falsely think you have not made any progress.

• your ultimate confidence boost will come, when a new learner asks you on how to do certain things and most probably that question is very similar to something you faced before. That can give you the boost of confidence that you were in a right path the whole time and the challenges you faced were all part of the journey.

Conclusion

This is the 2nd part of my original My journey to learn Python as a Petroleum Engineer. In the original post I shared the resources I recommend to a Petroleum Engineer to start learning Python based on my own experience. In this article, I address a challenge I saw many of my colleagues faced whilst trying to learn Python, losing motivation as they cannot see how far they have come or losing interest as they cannot translate learning to something to apply to their daily work. This article is my attempt to address both of these challenges.

ECLIPSE (ECL) has a lot of advantages over its competitor that made it industry standard. One simple advantage was the ability to use ‘INCLUDE’ files. INCLUDE files simply allow you to reference the input data in the main file, while your actual input can reside in another location. This prevents the main ‘deck’ from getting too busy.

That is great on a day-to-day operation, but sometimes you prefer to have all the data in one single file, particularly during troubleshooting an ECLIPSE error. $EXPAND macro that comes with ECLIPSE installation allows you to bring all the INCLUDES input data in to the main ‘deck’.

One day an admin right issue access meant I could not use the $EXPAND or any other ECL macro anymore, and as I was on my path to develop my Python skills, I tookthat as an opportunity(excuse) for development. I was also excited as the solution seemed benefited from implementing recursive functions.

Recursive functions

Let’s briefly review what a recursive function is before reviewing the code. You should be already familiar with a function in the context of programmnig is. A recursive function is special since it can calls itself within its own definition. In other words, a function that refers to itself in its own body.

When a recursive function is called, it breaks down the problem into smaller sub-problems and solves each of them recursively until a base case is reached. The base case is the condition in which the function does not call itself, and the recursion stops.

For example, let's consider the factorial function, which is commonly defined recursively. The factorial of a non-negative integer n is the product of all positive integers less than or equal to n. The recursive definition of the factorial function can be written as follows:

def factorial(n):    
    if n==0:
        return 1
    else:
        return n*factorial(n-1)

In this definition, the base case is when n=0, and the function simply returns 1. For all other values of n, the function calls itself with the argument n-1 and multiplies the result by n.

Recursive functions can be a powerful tool for solving problems that can be broken down into smaller sub-problems. However, it's important to ensure that the recursive function does eventually reach a base case, or else it will continue to call itself indefinitely and cause a stack overflow error.

Reviewing the code

Now let’s review the code to expand the whole DATA file and bringing all the include files information to the main deck. You can see the complete code in my GitHub repo, here.

First, let us initialise the code:

import os

file_name = 'DATA file name goes here'
path = r'The path to the folder containing DATA file goes here'
final_file = []

In this section, we import os library and give option to user to provide file_name, and path to the file.

Next step is not mandatory, but a nice section to have. Here the code checks if the file extension is *.DATA or if the user has provided a file with different extension. We are going to assume the file name is the same as the original DATA file, hence, changing the extension to match. This is a good practice to make the core more robust for different scenarios.

At the end of this block, file contains the full path to our *.DATA file.

if file_name[-5:] == '.DATA':
    pass
else:
    ext_ind = file_name.find('.')  # Making sure the DATA file has the correct extension

    if ext_ind > 0:
        file_name = file_name[:ext_ind] + '.DATA'
    else:
        file_name = file_name + '.DATA'

file = os.path.join(path, file_name)

Now we can create our recursive function as below:

def expander(data_file):

    flag = 0      # flags where code detects include files
    grid_flag = 0 # if grid files are needed set this number to non-zero
    
    include = os.path.join(path, data_file)
        
    with open(include, 'r') as inc:
        for item in inc:
            
            if item[:7] == 'INCLUDE':
                flag = 1
                continue
                    
            elif flag == 1 and grid_flag != 1 and item[:2] != "--" and item[0] != '\n':
                loc  = item.find('/')     # finding the path to the include
                item = item[:loc].strip() # removing spaces and new lines from the path
                item = item.strip("'")    # removing the qoutations from the path name
                expander(item) # recursive function if there is other includes within this include
                flag = 0       # flag back to zero so the lines can be writtern now
                continue
                        
            elif item[:5] in ['COORD', 'ZCORN'] and grid_flag == 0:
                grid_flag = 1
                continue
                
            elif item.strip() == "/" and grid_flag == 1:
                grid_flag = 0
                continue
            
            elif grid_flag != 1:
                final_file.append(item)
                        
    return final_file

The code has enough comment that you should be able to follow. but the idea here is we scan the *.DATA file, if we see any include file, we also scan that and attach it to the block of text in memory, with the exception of GRID section (optional) to reduce the volume of the text in the final file. If our include files have their own include files (mostly common in SCHEDULE section), our recursive function will call itself out and read them too.

Now that our function can read the file, we just need to simply, call our function and write it to a destination text file:

with open(os.path.join(path, file_name[:-5]) + '_EXPANDED.DATA', 'w+') as output:
    output.writelines(expander(file_name))

Finally, we can tell the user, the operation has been successful and where they should be able to find the expanded DATA file:

print('You can check the expanded DATA file in the following path: ')
print(os.path.join(path, file_name[:-5]) + '_EXPANDED.DATA')

I hope you enjoyed this tutorail. Feel free to drop me an email on hello@aminnoor.blog, or comment below if you have comments or questions.

One of the best life hacks I learnt was to create a dedicated music playlist for any trip that you go on. Later, when you listen to that music, they are associated with that vacation. My partner and I have made this a habit and expanded on it, we list to Podcast series or an Audiobook in our road trips and it has been great. Driving by and exploring the beautiful nature of Australia while learning about something new or getting entertain by an enjoyable book is a high pleasure that few things can reach that high for me.

As a reservoir engineer in an exploration team, an investor, and a Data science enthusiast I love dealing with probabilistic and uncertainties. Listening to the story of gamblers, seeing how they mastered the art of both applying probabilistic and using psychology to mark their names is a big attraction to me. In our latest road trip, we listened to season 2 of a great podcast called Gamblers.

I was introduced to Andrew Neeme, the Pied Piper of Poker, in an episode on the second season. Andrew is a professional poker player. He is not a WPT champion, nor is he famous for making hundreds of millions during his lifetime. What he is famous for is the innovative way he used his vlog. He is the first Poker player that took YouTube viewers behind the scenes of the life of a professional poker player. He achieved many things through that humble vlog adventure that you can listen to in the podcast, but what stood with me the most was through his vlog he made the life of a professional poker player accessible to all and that is a huge game changer.

If you want to take poker seriously as a profession, studying the life of the greatest of all-time may not be helpful to you. Those players are mostly tail events, they are right on the top, due to many exceptional circumstances that made them so special. Based on a normal distribution, you are far more likely to be somewhere in the middle than right on the tails. Hence, a skilled professional player showing you behind the scenes in Poker and showing you all the good, bad and the ugly can be much more relatable and useful when you want to start and decide if the path is the right path for you.

And that gave me the confidence to write more here. Instead of trying hard to write the ‘ultimate’ article every time, I will try to write more articles centered around my career and how I am building my skillset in Data science and Petroleum engineering. My idea here is, my geeky approach to various things in my career, might be as much fun, interesting and informative to others as it is to me and hopefully you will relate to some of it.

Going forward, you can expect 1 or 2 articles per month, ranging from how I am going to use Python to automate Reservoir Engineering workflows to how great my new EV is and everything in between. I will share my small discoveries (for the lack of better word). If I have a bright moment, an innovative idea, a new automation method that can scale, I will share it here and you will be the first to see it. I hope to see you in future articles. So, subscribe now if you haven’t already, so you do not miss my next writings.

Share

Over the last few years, I have tried to encourage my engineering colleagues to add Python scripting to their skillset. (if you want to start your journey I wrote a how-to here). Many of these colleagues and friends have sharpened their skills in Microsoft Excel or other propriety tools (OFM for example in Oil & Gas) for years. The thought of learning scripting in Python has been both exciting and scary for them. Generally, their arguments or their reluctance falls under one of the points below:

• Sunken cost argument (different excuses but all cantered around why should I learn a new tool, when I have spent so much time learning the current ones)

• My challenge/model is too complex for anything that Python can handle

• General fear of scripting which goes hand to hand with the next argument

• Anything you can do in python I can do in Excel (replace Excel with the software of your choice with Graphical User Interface (GUI))

In this article I am hoping to show you why basic knowledge of scripting in Python and a working knowledge of some of its libraries such as Pandas should be a must to have in any engineers’ toolbox.

Familiar Scenario

I have worked in my industry in different roles from an external consultant to the engineer responsible for delivering projects in operating companies and I have seen my share of useful but unreliable solutions developed on Excel and similar tools. The following story should sound familiar to you:

• Senior Engineer/Manager (M): We have plenty of data to work with, we need x number of more engineers to be able to create and run all these models

• Smart junior/grad engineer(E): Hey, it should be alright, I know advanced excel and VBA from my university days, I should be able to automate the model generation piece and we can do this within our team

• M: great, show me how your tool works, let’s present it to the higher-up management, we are great

So far it is all great, and it could have been great if the story would have finished here. As many of you may know, it does not. Time passes, and M thinks why not try to capture more complexities into the Excel Macro. As E starts to add more complexities, the Excel file starts to get larger and larger and then crashes start to occur. The file that used to open in a few seconds and run things fast, now takes minutes to open, half the time it is getting corrupted and the other half, it may crash in the middle of running the latest Macros. This part was common, but from this point on, I have seen different scenarios playing out in different companies:

• M and E decide to simplify their assumptions (sometimes to the point that they may not be valid, but the justification is we are constrained by the tool)

• M leverages business continuity argument and complains to the IT that IT should supply a more stable version of Microsoft Excel/Environment

• E suggests instead of running Microsoft Excel on one machine, why not running multiple instances on many, accepting 50% completion of macros over night as a success

Time goes by, Microsoft/software vendor updates the software and adds some improvements, but things barely get better, going back is not an option and scaling is a challenge no one knew they are going to face.

One may think the logical conclusion from the scenario above is for the engineering team to conclude they are using Excel in a way that is not suited and it is not the best tool for their problem. But unfortunately, my experience has been quite different, in at least two occasions the conclusion was our problem is so complex that no tool can really handle it!

The issue is sometimes we tend to focus on our field of expertise the most and forget that many other industries have very similar issues like us and they may have developed solutions that we can borrow from. There are on average 1 billion tweets sent every 2 days, if twitter can handle that and do all sort of analytics on it, there should be tools capable of holding 400 simulation models for 1000 wells for 40 years. The great news is, the solution exists, it is free and it has one of the best communities in the world supporting it. It is called Python and all it requires from you is to spend some time to learn its language.

Why YOU should treasure Python over Excels of the world

If you have not been convinced yet (which if I know my engineer friends and colleagues, the chances are highly likely that you have not), below I am making a case on why using Python to play with your data is a much more superior tool compared to Excel or any other Proprietary tool.

It is also worth mentioning that I am a daily user of Microsoft Excel and use many other tools which are not open source or have Graphical User Interfaces (GUI). My arguments here are to persuade you that the initial cost of learning Python can pay many years of dividends and a huge bargain. My mains arguments are:

• We should use more Open Source and less Proprietary software

• Scripts are powerful. They can eat GUI for breakfast any day

• There are future opportunities unforeseen today if you are not confined by your tools

Let me explain them in a bit more detail:

1) Use more Open-Source and less Proprietary software

IBM defines open-source software as:

Open-source software (OSS) is a decentralized development model that distributes source code publicly for open collaboration and peer production known as “the open-source way.

What that means in day-to-day practice is for the Open-Source Software the source code is open to be seen and changed by anyone. So, if your action causes the software to crash, you can try and find the bug, and fix it for yourself and also share it with the world. The next version of the software has the fix you made.

It may not sound hugely important if you are a beginner in python and you are only going to use Python for automating some mundane tasks, but think about some irritating issues in some of the popular software you use that haven’t been fixed for years, You can check this thread on challenges of working with large Excel datasets that Excel power users are facing or this issue a user has faced for years. The user request does not get priority because it is not common, but for the user who has the issue, it is an everyday pain. You are dealing with a big corporation like Microsoft with a global user based reach. They have to prioritise their resources on what issue to address, and if your painful issue is not common then tough luck.

2) Scripts can eat GUI for breakfast

Here I am going to give you two reasons on the superiority of scripts over GUI:

• Engineering work requires a lot of mental power and focus. You cannot get interrupted constantly and still produce quality work. Working with scripts rather than GUIs seems scary at the beginning. As you get more comfortable with writing scripts and navigate your way through things using text commands and scripts, you realise staying in a single environment minimises the distractions. This ability to focus more can lead to generating higher quality work. Using mouse and GUI while visually enticing would cause you to move from one screen to the next which means your eyes glance on various things on screen. They can distract you and keeping yourself focus drains your mental strength.

• The second importance of the script vs GUI is the reproducibility of your work. There is a learning curve. It may make you slower than the rest at the start.As you get comfortable though, you will spend less time on the preparation and more on analysing and engineering while staying focus for longer. Imagine a simple case of copy/pasting a file. Compare automating or repeating a simple file copy/paste. The movement of the mouse, right clicking and selecting copy, going to the destination to paste the data, vs the following (bash not Python):

                cp SourcePath/file DestinationPath/file

3) Unlocking Future opportunities unforeseen today

When Steve jobs introduced iPhone in 2008, Steve Balmer, the Microsoft CEO at the time, not only laughed at the question on iPhone but also described iPhone as “the most expensive phone in the world”, one that “doesn’t appeal to business customers.” What many could not see at the time was how iPhone would profoundly change how we interact with the technology and the world. Who could have envisioned we would buy or discuss ‘Bored ape” pictures (current floor price is $150K USD) in less than 15 years after iPhone launch using our phones. The same goes with Python and our day-to-day engineering challenges. In the current Excel driven mindset, the possibilities are confined with the tools at hand, but with a different toolset and when the possibilities are almost endless, who knows what applications each engineer can think of. I am excited to see those, aren’t you?

Final words

In this article I showed general scenarios on how engineers can end up with closed source solutions and run into scalability issue as the challenges are getting more complex and the datasets are getting larger. I offered my solution as adding Python to our toolbox.

If a task worth doing, it is worth doing it right, and if you think you need to do it more than once, try to think of automating it. One of the best tools to help you in that path is Python and its wonderful community. Python is free, open source, easy to use in automation projects, extremely flexible, has been tried across many industries and disciplines and is constantly evolving.

If you want to see where to start on your Python journey, I wrote about it here:

My journey to learn Python as a Petroleum Engineer
I try to answer 6 commonly asked questions for new users on learning Pythonaedalat.medium.com

SPE’s intorudction of the PetroBriefs series is as follow:

“it is meant to quickly bring each reader up-to-speed on an emerging technology or specialized topic. At around 100 pages, most are available in both softcover and in popular eBook formats.”

I downloaded the book yesterday and read through it using Adobe Digital Edition. (if you have not used Adobe Digital Edition, prepare yourself for a poor user experience). The list of the authors and reviewers of the book is very promising so is the list of the contents. I will try to make my review short and to the point by first quickly mentioning the pros followed by cons.

Pros:

The book is brief but covers many of the essential topics. In this regard, the book provides the reader with a very good fundamental understanding of the current challenges and opportunities in data analytics in the Oil and Gas industry. I, for one, learnt that the challenges I am facing in data preparation are more common than I thought they are.

Chapters and sections are independent. Many of the chapters start with some introduction on basic concepts required for the chapter, even though they may have briefly discussed elsewhere in the book. This made it very easy to start reading from the topics/chapters you are interested in without the need to read the rest of the book.

No jargons. If I want to borrow from Cassie Kozyrkov, the authors were ‘data scientists’ not ‘data charlatans’ in a sense that they did not allow the catchy phrases and topics get in the way of their judgment. They were very informative on the current state of analytics in the industry, where the industry is behind and where the industry and Petroleum engineers should head or aim for.

Reference list. There is a great reference list at the end of the book which provides a fantastic opportunity for further reading. We are still at the beginning phases of the adaptation of data analytics in the Oil and Gas industry and there are many avenues for further reading and research.

While data analytics has broad applications in reservoir engineering, the vast number of wells and pace of operations in unconventional allow data to play a critical role in the decisions that create value.

Cons:

Adobe Digital Edition: It is the only medium the ebook is available on and my experience with the platform is very poor.

Scalability: I understand it is a PetroBrief, but I wish the book would have tied up in a bigger series and would have been used as a stepping stone to reading other books which could expand on its topics. I also understand this can be done by the readers and chapters come with good references for future reading, but I feel it is a missed opportunity for the authors not to provide a suggested path (maybe at the end of each chapter) to suggest further reading material for the enthusiasts among us. Maybe there is room for a more detailed book and this is what I am looking for.

For those who,like me, are working in unconventional assets, the book has a promising message. “While data analytics has broad applications in reservoir engineering, the vast number of wells and pace of operations in unconventional allow data to play a critical role in the decisions that create value.”

I highly recommend the book for those who want to start their data analytics journey or the managers(decision-makers) in the industry who wish to understand the latest trend and equip themselves with the required knowledge required for decisions of tomorrow!

To be called literate in the 2020s there is a good chance you must know how to code. It may seem an exaggeration, but I certainly believe it would be true for engineering roles. I still remember vividly when in 2014 President Obama participated in the ‘Hour of Code’ to encourage students to pick up coding. Those students are entering the job market now and in the coming years and I cannot even imagine how much of impact they would have in the way we live and work. One thing I know is learning how to code is fun, liberating and can save you lots of trouble in the long run while makes you look smarter than you are (certainly true in my case).

Now if you want to start learning Python, you do not have a problem of finding resources, you will have a unique problem of picking the right one. Sometimes it seems to me there are more python tutorials online than the population of earth! (a bit exaggeration but you get my point). My experience in learning new skills over the last few years has taught me one thing, the most time-consuming yet crucial part is mapping the learning path. If I get that part right, things fall in place nicely. I spent a lot of time online and offline discussing with experienced users to map the path for my learning and over the last few years, I have helped many friends and colleagues with where to start Python and how to approach it. In my previous workplace, I advocated for a formal python training to leadership, once I got their approval I researched, picked and tailored the course to suits our Petro-technical engineers and the course was successfully delivered on February 2019.

Today I am going to share with you some of the frequent questions I receive and my answers to them. My hope is these questions and answers would ease your way on your journey to learn Python. I share the post on my LinkedIn and my Medium page, and I hope to update it regularly. Please use these as a guideline alongside your research. If you find alternative resources that were helpful please feel free to share them with me or comment them down below for everyone’s benefit. Also please do not hesitate to ask away your questions in the comment or direct message, I would be happy if I can help.

Question 1: Python or R, which one should I learn?

If you are an engineer or want to code for an engineering solution, my answer is Python. Python is the second most popular programming language now while R is 14th and they were in similar rank not long ago. You can read about some of their differences here. As a beginner for at least the first year of learning python, there is a very good chance that any problem you face has an answer ready for you on the internet and that makes the learning process a smoother journey.

Question 2: Python2 or Python3?

Python 2 is dead. So, if you are about to start learning Python, don’t even consider this question as part of your research.

Question 3: Where to start learning Python?

I picked up a few different courses, to begin with. As an engineer, I work with excel sheets, PDF files, office documents every day. Why should I pick up a course that teaches python very well, but the projects are about tic-tac-toe and some other random games? This is why “Automate Boring Stuff with Python” is my top recommendation. Al’s book and delivery are great. More importantly, the course is very practical which allows you to start coding on your small projects very quickly. Those small wins hopefully are going to motivate you and make it easy for you to commit to learning.

Question 4: I finished “Automate Boring Stuff with Python”, now what?

If writing scripts is all you want to know from Python, then “Automate Boring Stuff with Python” should be enough.

If you want to learn python more in-depth or you are thinking on developing applications with more complexity than automating scripts and such, then you may want to have a look at a computer science course with a focus on Python. I highly recommend the 2-part series “Introduction to Computer Science and Programming Using Python” by MITx which is available through the MIT website and Edx. What you get out of the course is how to frame your mind to code like a computer scientist. It is invaluable in making you a more efficient programmer and more comfortable with Python and its libraries’ documentation.

Question 5: Is there any other resources to learn from.

Yes, plenty, I list them here and try to explain in a few short sentences why they have been shortlisted and worth your time:

Think Python:

Great book for an introduction to Python, available for free. What I liked the most was how each chapter has some exercises to help you to judge your understanding.

A whirlwind Tour of Python:

It is a fast-paced introduction to Python and it is tailored to those who are new to Python but have programming background in other languages. If you are profieceint with VBA and you want to pick up Python, maybe this would be a good starting point for you. It is available for free from various sources, such as here and here.

Python for Data Analysis:

If you are ready to learn pandas, then why not learning it from the person who created it. Wes’s book is the go-to for learning pandas, combined that with his videos on YouTube and then learning pandas would be fun. You can purchase the book here.

Medium:

I found Medium website and “towards data science” publishing page particularly useful for finding like-minded people, latest trends and general coding/python tips and tricks that would be handy.

Reddit:

Reddit needs no introduction. I found learnpython and datascience subreddits very useful forums to follow.

Stackoverflow:

Last but not least is stackoverflow. There is a very good chance that any python questions that come to your mind have already been answered here. So, you would learn to rely on this very early in your learning journey.

Question 6: How can I practice.

Try to find easy projects around you. I know this is easier said than done. Finding a good personal project would be one the most challenging part to you in this journey as I had a hard time finding a project that worth doing, yet can deduce to simple challenges so it fits in my skill set and I can approach it.

If you cannot think of any projects early on, don’t panic, it is as natural as losing breath after running a marathon( in my case around the block). The solution is codewars. I found codewars around 2 years ago and that helped me solidify many of my learnings. It gives you small projects to practice your python skills, try to start from basics and gradually increase the level of difficulties.

I have developed a few small and big projects with Python over the last few years, I share two of my favourites below to hopefully gives you some motivation and idea.

The first one is a personal project I did at home using Raspberry Pi and my Solar system. My python code requests the energy output of my solar system every minute, collects it in a database and plots them for me on my command. I can check how many sunny days I had in a month and monitor the quality of my panels over time.

The second one is work-related and it is a project I did at my current role. My application successfully reduced the time required to generate gas and water type curves from over a week for our big fields to minutes, by automating a lot of calculations which were previously run in Excel.