How we built a machine learning model

How do you develop and successfully implement a machine learning model as a small company with minimal resources? This is the story of how BetterNow developed and implemented a machine learning model - 'The Fundraiser Coach' - which helps P2P fundraisers do their very best.

The idea

On the surface, it seems like a rather stupid idea: Develop and implement a machine learning model as a small company and still have the operations run smoothly, including handling millions in donations every month. 

Luckily, no one at BetterNow has ever been scared of a challenge. On the contrary, the chairman of our board - Kim Tosti - has long been very bullish on the idea that BetterNow should employ AI to make the world more generous. And his opinion carries a certain weight as he also is a consultant with 2021.ai and has recently taken courses in Machine Learning at the Danish Technical University.

We knew we had a lot of data, but where would it be relevant to apply machine learning? So we took an exploratory meeting with a couple of data scientists at 2021.ai. We came with a few of our own ideas, and quite fast, we narrowed in on the idea of using AI to generate recommendations for individual P2P fundraisers to improve how much they fundraise. Thus, the idea behind the fundraiser coach was born.

2021.ai agreed to some preliminary analysis on a prepared set of data from us. The results from this analysis led us to believe that there was a rather substantial potential here. We could see that certain variables did have a significant impact on the outcome of fundraiser success. In other words, the successful development of a P2P fundraiser wasn’t random in our model. The data we had could explain part of the success. 

We name this project ‘The fundraiser Coach’ as it would function like a coach for each individual P2P fundraiser, giving advice and tips based on their indicidaul performance and profile. 

This project was perfect for us. A key promise of our solutions is that charities fundraise more by using it than by building something themself or using a competing solution or platform. Our mission is to make the world more generous, and this project also fits perfectly with this goal. 

Funding the project

But could we do this? We concluded that we had the internal resources to implement it and probably, most importantly - understand it. Matt, CTO of BetterNow, had worked with machine learning algorithms before. I have had enough training in econometrics at university to understand and work with data scientists.  But we needed help with doing the data analysis, and data scientists don’t come for free.

We had previously considered applying for a grant from Innovation Foundation Denmark, but we never had the right project. But this project was almost perfect. We pulled together an application based on the preliminary results and a project plan. We added a video and found a few of our customers who agreed with us on the potential, and we ended up with an application good enough for us to receive the full grant. This process also has given us a newfound appreciation for grant fundraisers!

Innovation Fund Denmark is speedy at processing applications and also have a very smooth process for applying. So not more than a few months after that first meeting, we had a project, funding and timetable!

The data analysis

The first part of the data analysis was primarily centred around editing and improving the data set. The task also includes ensuring that the data didn’t include personal information and planning how we later will put the model into production.  Another part of this task involved hours of manually categorising old fundraisers into various aggregated buckets; not all machine learning work is fun! Then a lot of tests were carried out on the final dataset, and we removed outliers.

Developing the model was a very iterative process. We constantly tried new approaches, played with new variables and investigated partial effects. We, for example, ended up doing some quite advanced image analysis on profile images. The results were very informative, but we did not include it in the final model, as image processing is server (and energy!) intensive. We did this as we could see that the type of profile image only had a limited impact on the fundraisers.

The data analysis, in my opinion, is the most exciting part of the process. We have gained a tremendous amount of insights into how fundraisers and supporters use our solution. We have found what works for fundraisers, spotted patterns we never would have noticed otherwise and learned much more about our users. We only included a fraction of these learnings in the final model, but this learning might be just as beneficial to us. This is an essential lesson for anyone considering working with this, that half the benefit might come from the process itself.

Implementing the machine learning model

Implementing the model took a lot of research and preparation as we had to consider several tradeoffs.

1. First, we have privacy and security concerns. We don’t want the model to use any personal data. And we don’t want any security flaws in the model where we inadvertently open any possibility of losing data or accessing it by unauthorised third parties. 
   ‍
2. Secondly, speed is a concern. Running a  machine learning model can easily take so much time that results aren’t available in real-time. The way to increase speed is by using more energy and resources, which we obviously don’t want. BetterNow is Co2 neutral through carbon offsets, but we want to offset as little carbon as possible. 
   ‍
3. The third constraint is maintenance cost, especially the ease and time of how we re-train the model.

Our implementation model weighs privacy, security and low energy above speed and ease of training. These choices mean we have ended up with a model that isn’t real-time, is somewhat challenging to re-train, but does not access personal data (except when it is re-trained) and also doesn’t spend much energy (also lowers our hosting cost).

Defining and implementing the user interface 

Defining the Ui is something that we did simultaneously with the implementation. We decided not to have the model operate in real-time, and therefore we also can’t rely on a user interface where recommendations are updated while accessing the site. Instead, we generate recommendations for the fundraiser at various intervals and then present these to the user. Thus, we here rely on two different user interfaces. 

First, we have a newly developed section for each fundraiser called ‘fundraiser tools’. Here we present the recommendations that the model deems to have the most significant impact if followed. We have also added a tool for thanking donors, sharing options, and downloadable materials from the charity. This way, we ensure several reasons to visit the page and thus be presented with the recommendation.

Second, we email the recommendation to the fundraiser. We do this every time it is re-calculated through the model. Email is a meaningful way to reach those fundraisers that rarely log in.

Built into the model is logging to see if fundraisers see, read and follow the recommendations. This data is vital information for when we will re-train the mode, and we expect that the model will improve by a lot, especially the first few iterations.

Testing and launching

We had the model running without releasing the user interface for several months while we tested and looked for consistency and sanity in the recommendations. And it is safe to say that we have caught several weird results and added various conditions ensuring users won’t get inexplicable recommendations.

During this period, we also worked to ensure that the model is in line with our ideas about data ethics and data privacy (link to data ethics policy).

The launch was pretty low-key, and not much happened. We launched initially with only showing the recommendations in the dashboard and then later added the more ‘noisy’ email notifications. This result is precisely what we hoped for. 

Next steps and the future

Now we are waiting for the results to check-in. We will monitor the difference in the success of fundraisers created after the launch of our model compared to before. We want a decent amount of data to calculate this, so we will probably wait a few months yet befor publishing anything. 

Note: The results are in, a 40% increase in the average fundraised per fundraiser

The first re-training of the model will be especially interesting, as it will add a lot of new data to the model. This data will be on how fundraisers have acted on past recommendations, what actions they take after seeing it, where they see them and what the fundraiser results are afterwards.