Egg Farmers of Canada entrusts CRG to unscramble its data for forecasting price trends

Soma Tah

Eggs are a staple food all over the world and one of the inexpensive protein sources, but there's a lot involved in the production to plate impacting the pricing of eggs. Egg prices generally get determined by keeping egg production, feed costs, climate, seasonality, egg export and shipment schedules in mind. With so many variables driving the demand and pricing, an understanding of the drivers of demand, costs and prices and a price forecasting mechanism can certainly come in handy, but that depends on the collection and analysis of the right data in a timely manner.

Things that keep the Egg Farmers in the dark

Egg Farmers of Canada (EFC), a leading agriculture non-profit in Canada was aware of this challenge and therefore wanted to evolve their processes that analyze complex pieces of data quickly to make the right decisions that would optimize their planning. EFC manages the country’s supply of quality eggs and also develops a national standard for egg farming.

In addition to managing the country’s supply of eggs, EFC also manages the supply of eggs to industrial egg product processing plants through the EFC Industrial Product Program. It’s a lot of eggs and a lot of data that EFC had to manage. 

The paper-based data collection was a slow and labor-intensive process. Every pallet of eggs that enters a processing plant gets tracked for weight, sizes, and quality. Incidents that result in lost or damaged eggs, as well as defects, are tracked. The movement of eggs throughout the supply chain also get recorded.  This data is crucial to EFC’s mandate to efficiently manage the national egg supply and maintain standards of quality.

Generating reports for provincial egg boards, industrial product processors, egg farmers and the general public sometimes used to take weeks. It was practically impossible to track the movement of eggs through the supply chain in real-time. Hence, EFC decided to automate this process.

Navigating the challenge

While they were evaluating solutions that leverage deep learning models to forecast egg prices. EFC had already been using Tableau as its main data visualization platform and decided to invest in artificial intelligence and data learning models using the Tableau platform as its first step before trying to include all these variables as variates or co-variates in the prediction models. Tableau was a natural choice, as it met EFC’s key criteria which included ease of use for the average business user, strong product support and easy upgrades. The graphical visualization and dashboards are designed to help organizations see and understand their data, and is ideal for generating detailed operational reports, on demand.

EFC turned to its longstanding business solutions partner, Corporate Renaissance Group (CRGroup) to do the customization and implementation of Tableau to ensure it served the unique needs of EFC’s operations. CRG Solutions is a global solution provider helping organizations navigate their Data to Insights journey, and specializes in improving enterprise performance through innovative technology-driven solutions.

EFC CIO, Tom Borowiecki asked the CRG to use different models such as LSTM, Prophet, Neural Prophet, SARIMAX, and Garch to improve their learnings when forecasting egg prices and to design an end to end automation process to ensure that no manual intervention would be needed to:

a) Execute the models periodically

b) Store forecasted values of each run date for future reference

c) Generate forecasting efficacy report with comparison of actual vs. forecast

"The CRG Solutions Data Science team used the TabPy extension of Tableau to utilize most advanced deep learning techniques of Python Libraries and run on the fly and display the results using Tableau Visualization. It was a very interesting solution, with the customer needing minimal manual efforts & intervention,” said Ravi Gupta, Program Head- Advanced Analytics, CRG Solutions.

While Tableau was used for a visual representation, the CRG team harnessed Python libraries using deep learning techniques to generate accurate forecast. The following components were used to achieve end-to-end automation:

-Tableau for extracting historical data

-TabPy to connect & run the Python program

-Leveraging Python libraries like FB Prophet, LSTM, Neural Prophet, Sarimax & Garch etc.

-TABCMD Command line extension to read forecasted values and save in a file

-Tableau Prep to read the file and populate structured Snowflakes data base table.

The deployment approach was designed to ensure zero manual efforts to execute the models and save the forecasted values for efficacy measurements.

Transforming data into action 

By using Tableau effectively, EFC got rid of all the manual processes and was also able to reduce the massive amount of time and effort that used to go into collect, verify and manage the data. Reports that took weeks to generate earlier were a few clicks away now. That allowed EFC to spend more time interpreting the trends and making more timely and informed decisions to more efficiently manage the supply chain, and also brought transparency, reduced errors and helped to eliminate the possibilities of any manipulation of data.

The end-to-end automated forecasting model helped EFC with the following things:

-Accurate forecasting results for better planning

-Comparison of actual vs. forecast in a visual representation

-Flexibility by having input parameters to choose forecast length

-Ability to adopt to new conditions

-Ability to include other variates to further improve forecasting accuracy

CRG Solutions team was able to deliver the completed solution in record time over 10 weeks. “CRG has delivered on time and with quality– a capability to us that not only integrates well with our existing process and tools, but also provides us with advanced insights that will help us continuously improve how we run our business,” said Borowiecki.

Later, EFC went on to extend Tableau to automate and streamline processes across other departments too. Thanks to Tableau’s easy-to-create dashboarding and filtering tools, the same data of EFC is now being used to support a more user-friendly and timely market information website for the public. People can visit the site and easily carry out customized searches through years of data with a few clicks of a mouse, without EFC being required to go through a multitude of expensive custom upgrades to a legacy system. Tableau is also being used to support the data collection for EFC’s national on-farm food safety program, Start Clean—Stay Clean, and Animal Care Program.