Data in the Wild

[00:00:00] Ola: Welcome to Data in the Wild, a podcast from Xata, the serverless data platform where we talk to indie hackers about their real world experiences with modeling data for their products.

[00:00:14] Benedicte: And today we are talking with Monica Lent, the founder of Affilimate. Welcome to the show, Monica.

[00:00:21] Monica: Thanks for having me, team.

[00:00:23] Benedicte: So Monica, what is Affilimate and what problem does it solve?

[00:00:27] Monica: So our product kind of breaks down into three main parts. So what we do is we basically do data aggregation of affiliate revenue for media companies and creators.

[00:00:37] So they have affiliate commissions and a dozen different platforms. We bring them all together and then enrich those commissions with data they can use to optimize their revenue.

[00:00:48] And then provide some tools for their teams to kind of scale their commerce content workflow.

[00:00:53] So yeah, we have kind of like two main data types that are really interesting. There's like the transaction data, obviously it involves money. So that makes it interesting. And then we have the web analytics data. Oh, which is where we are. Yeah.

[00:01:08] We're tracking web analytics like page views, clicks, impressions, et cetera, on our publishers' websites.

[00:01:15] So yeah, those would be our two main data models.

[00:01:19] Benedicte: Ooh. But so before we get to more in depth about data modeling for Affilimate, could you just run us through the tech stack?

[00:01:27] So the listener isn't like, I can imagine them being like, "but where, but where?"

[00:01:31] Monica: Yeah. I mean. Like we're on GCP as we have established earlier. So everything is on Google. And a lot of our user data is inside Firebase.

[00:01:44] So everything that is just like settings, permissions, et cetera. But we have a whole medley of databases, including BigQuery, Postgres, and Redis.

[00:01:54] And then, yeah. Again, it's all serverless. So we're using cloud functions for everything else, which is great because it scales, but also not great because it scales.

[00:02:05] In terms of your budget.

[00:02:07] So yeah. I would say that's kind of the highlights.

[00:02:11] Ola: In your time with modeling data, have you come over like a gruesome experience that you would like to share with us or like a tall tale?

[00:02:23] Monica: It has to be gruesome?

[00:02:24] Ola: No. Well, actually the definition of a tall tale, and I do this all the time when I'm at the pub, is you tell a story like it's you, that's like the hero of the story. But it didn't actually happen to somebody else.

[00:02:40] Monica: Oh. Okay. Oh my God. Okay. I mean, maybe I can tell my tall tale in a way that this tall tale was gruesome to my bank account.

[00:02:54] So yeah. I mean, a couple of weeks ago, maybe a month or so ago, I was in bed. It was almost midnight. Or maybe even past midnight. And I got an alert on my phone that said, you have reached 100 percent of your cloud budget. Which is not something you want to receive like a week into the month. And so I was like, "okay, this is bad."

[00:03:23] So I'm like, I get out of bed, you know, kind of crawl my way over to the office that I'm sitting here today. Turn on a lamp, open my computer, and start looking at, you know, all of the monitoring and dashboards. And you can just see there is this line that was flat, and then it went like this, and then it was flat like that.

[00:03:46] Like, it just went up, and I was like, "no!"

[00:03:51] And so obviously, that was a bit stressful because, you know, our budget, I mean, we spend quite a bit on cloud billing every month, or like on the cloud every month. Like, it's a lot of money, and so multiplying that potentially is a scary thought.

[00:04:09] Short version of the story is that I identified which cloud function was responsible for this. And what happened was that I had a queue running. But unfortunately, cloud functions that fail don't report back in any way to the end of a, you know, they don't have a way to signal that they failed.

[00:04:27] And so what happens is that this queue gets stuck into an infinite loop, and it's using a lot of resources when it does that.

[00:04:34] Benedicte: Yeah!

[00:04:35] Monica: I had to just delete the cloud function because it was not like essential for like our tracking infrastructure. It's like a background job that's obviously like running overnight. So that was like, not too bad.

[00:04:46] I was able to delete it, you know, within maybe 30, 40 minutes. But the problem is that billing analytics are not real time.

[00:04:55] So even after I deleted this cloud function, the bill went up still by like hundreds and hundreds of euros like, you know, through the next morning. And I couldn't feel confident that I actually solved the problem, you know, because the bill just kept increasing.

[00:05:11] And eventually it stopped. And what I learned is that anytime you implement some kind of a cue, you need to also record like number of retries that are allowed and always increment those retries before attempting to execute. So now we have other cues that work that way.

[00:05:29] But thankfully I was also able to appeal to cloud billing support and get back a 90 percent refund.

[00:05:37] So thank you. Yeah. For that mistake.

[00:05:41] I mean, we spend a fair amount of money with them anyways, and they just gave it to us as a credit. So it's like, "okay, they're going to get these thousands of euros soon enough.

[00:05:52] So pro tip, you know, if you mess up something on GCP or AWS and you are already, yeah, spending, you have a bit of a history with them, then you can potentially get that appeal as long as you don't make the same mistake too often.

[00:06:07] Benedicte: That is so great though that you could get that they would actually like listen and like give you some of that back.

[00:06:14] Monica: It's not easy to get in touch with billing support. You have to definitely jump through some hoops. At the same time, I was also trying to buy like a more advanced support package, but this involves some like migration permissions and stuff, so it was not possible.

[00:06:28] I just had to like go delete some stuff and make it stop.

[00:06:31] Benedicte: Yeah, I think the Norwegian developer magazine, Kode24, which is like Code24. They had an interview with some guy who had like a side project on AWS and like his credit card bill was like, I think it was like $6,000 or something like that.

[00:06:48] And I don't know exactly how that got resolved, but I was like.

[00:06:52] Monica: It's so easy to accidentally spend a lot of money on the cloud.

[00:06:56] Like I once deleted something from big query and it costs 300 and that was like early on in the project. So like if we were to delete something like that now, it would all, it would probably,

[00:07:07] I mean, I could probably calculate how much it would cost, but it would be thousands and thousands of dollars, like,

[00:07:13] So, yeah.

[00:07:15] Benedicte: Well, that was a great tall tale. Thank you for kicking off the tall tale.

[00:07:19] Are you happy with the tall tale, Ola?

[00:07:21] Ola: Yes.

[00:07:21] Benedicte: Was it too spec?

[00:07:23] Ola: Yes.

[00:07:25] Benedicte: So we found a little rant on Twitter. Do you want to read that, Ola, and ask Monica?

[00:07:31] Ola: It's about user permissions and roles.

[00:07:33] So it's wild how many SaaS tools out there don't have a way to model many to many relationships the way business to business SaaS demands.

[00:07:46] For example, users versus organizations. This pain exists in every product we're using from email to CRM and now product analytics. And then it's like these clouds out of the nodes. Yeah.

[00:08:00] So could you walk us through the steps you have taken to get this data model right in your product.

[00:08:07] Monica: Yeah, of course.

[00:08:08] I mean, it's funny because a lot of these products, they, since they are built for B2B SaaS, they are handling this kind of user and organization behavior on their side.

[00:08:19] But then as soon as they offer a SaaS product to us, it's like not fully thought through how we're supposed to bring in our own user and organizational data into their platform. Which just doesn't really make much sense.

[00:08:33] But from our side, I mean, you know, we had from the very beginning, the distinction between, workspaces and users. So all of the account data. is basically connected to a space, and then we also have separate user data and the users can belong to multiple spaces.

[00:08:54] So that's something we got right from the beginning.

[00:08:56] Ola: But maybe that's the reason why they don't do it because it's like, "Oh no, we have to do it now," like after the fact and it's hard.

[00:09:03] Monica: Yeah. I think people think it's an add-on later maybe. But it's definitely worth it to do from the beginning.

[00:09:09] Ola: But no, I was saying that maybe it's hard to do it later because you already have like

[00:09:14] all these things set up and

[00:09:17] Monica: Oh yeah. I mean, data migrations are not fun no matter what. So that's true.

[00:09:23] Benedicte: Yeah. So you had that set up from the beginning with users. And users could be part of a workspace.

[00:09:31] Yeah. And going from there, so you had that from the beginning, great!

[00:09:37] Monica: Mm hmm.

[00:09:38] Benedicte: Was it perfect for ever and ever?

[00:09:40] Monica: Of course not. There are some things where, you know, we attached certain types of settings to the workspace. Where they should have been attached to the user, like per user per workspace, which gets a little bit confusing.

[00:09:54] And then what, like a big project that I've been working on recently was the user roles and permissions. So we started out with very like high level user roles and permissions, and now we have more than like 50 kind of distinct types of scopes that then can be combined and bundled together into different types of roles.

[00:10:17] So, yeah. We started with something as simple as, "oh, you just have a data type, and then you might need to create, read, update, delete." But of course, like, apps are a lot more dynamic than that, and that's not, like, fine grained enough. It doesn't, like, reflect the user interface or the behavior of, like, what people really need to be able to do.

[00:10:37] So that was a modification that I have been working on most recently.

[00:10:43] Benedicte: So some things that I see when I try, when I've used products is that they kind of have this in place, but then I'm not allowed to be part of multiple organizations. Like I am part of an organization, but I can't be a part of multiple organizations with the same email, which gets quite hard if you're a consultant or, you know.

[00:11:04] As a developer, I can work my way around that by just having an unlimited set of Google accounts, where you just do like, you know, the Raae plus something before the rest of your email account.

[00:11:19] But yeah, so that's, is that one of the things that you had in place where a user could be part of multiple organizations, like a virtual assistant or like a marketing consultant?

[00:11:29] Monica: Yeah, I mean, we have the concept of workspace, and so a user has always been able to be part of multiple workspaces.

[00:11:36] Benedicte: Oh, because a company could have multiple workspaces.

[00:11:39] Monica: Yeah, yeah.

[00:11:40] And then also if they have, let's say they're a contractor or something like that, then they could be invited to workspaces owned by different organizations.

[00:11:53] But we don't actually currently have like a data concept of organization that is like the parent to a workspace.

[00:12:01] So, but at the moment I don't think we really need that. I don't know. We will have to see if there's like for some reason that they really want to keep the data separate. It's like a big organization that has multiple workspaces.

[00:12:14] But for the most part, all the data for an organization goes into a single workspace because they want to be able to like compare the data to one another. So separating it out would not make too much sense.

[00:12:26] Benedicte: So what made you do the lift now of changing the kind of the roles and user permission that you had in place?

[00:12:33] Like what was the, the kind of thing that was like, "okay, now we need to upgrade this, now we need to change this."

[00:12:39] Monica: Yeah. I mean, it's just that we start to work with more and more complex organizations that have different needs in terms of what should and shouldn't be visible to the end users.

[00:12:50] And also because we have a tracking system, you know. You maybe don't want to add like the junior content editor with the ability to like potentially mess up the integrations in the account, even if it would just be a mistake.

[00:13:04] So you want to kind of be able to say, "okay, like these things should be disabled or we don't want to show these things." and so now we have like really fine grained ability to like offer predefined roles for, you know, on some of our pricing plans.

[00:13:19] And then for like enterprise or like larger customers that they are able to build, like very custom roles that are just like very, very fine grain on kind of like a data type and feature level to kind of hone down exactly what kind of visibility and user permissions they want to grant to a type of user.

[00:13:41] Benedicte: So the person from the company, the owning company of the workspace, can add a user and then kind of just check boxes for the different things that that user should be allowed to do?

[00:13:51] Monica: That's what it's going to look like once I build the user interface.

[00:13:55] Benedicte: Oh, you have the features in place.

[00:13:58] Monica: Yeah, it's all, it's all implemented. Yeah, it's like. It's implemented in the, in a way that building that interface will be possible.

[00:14:06] Upfront, what we'll probably do is that as companies need this, we will probably start by consulting them a little bit about what they need, because sometimes, like, let's say you add access to one permission, but not another. But these things maybe kind of interdepend on each other.

[00:14:24] And so it's a little bit easier to take a consultative approach and say, "okay, all right, you have this use case. Maybe it's someone from your finance team. And finance needs to be able to see this revenue data. But you don't want them to change the tracking settings. Okay, this is the exact visibility that we can recommend that you give them to achieve kind of what you're after."

[00:14:45] So it's built in a way that all the predefined roles are modeled in an identical way that the user defined roles are going to be. So that way it's all kind of like interchangeable and it's going to make it easy for us to like add new roles.

[00:15:00] Like it's literally just give it a name, combine some scopes, and you're good to go.

[00:15:05] Benedicte: Yeah. Because that was going to be my next question. So scope would be then kind of in my head, that would be each of these checkboxes. That's what you would call a scope?

[00:15:14] Monica: Yeah. Yeah. Something like that.

[00:15:17] Benedicte: That would be the scope. And then permission is then giving a user that particular scope.

[00:15:23] Monica: Yeah. So we think about permissions as basically being a bundle of scopes with a name. And then when you give a user one or more roles. And a role is a bundle of scopes, right? Then you can also add additional scopes.

[00:15:42] So let's say you just want finance team to be able to have the finance role, but then you have this one person who needs access to one extra feature. You don't want to have to define an entire additional role.

[00:15:54] So in that case, you would be able to say, "okay, this goes to the finance role. And then this one user has this additional scope as part of their permission set."

[00:16:04] Benedicte: Oh, so these are then just to be very nitty gritty. Yeah. Cause you're teaching me here, you know.

[00:16:10] So whenever, if I'm ever going to go out and do something like this, then just to be clear, so then you're doing this in Firebase, right?

[00:16:18] Monica: Yeah. All of the permissions exist inside Firebase.

[00:16:23] Benedicte: So then you kind of have a permissions or a scope table and a role table. And a permissions table? Or am I getting too detailed now, maybe?

[00:16:34] Monica: So for the, for. It's almost there.

[00:16:37] So like for the default roles that we define, we define those in code. So that way it's easy where if we add new features for the default roles, we can automatically grant access to those features because that's becomes a bit of a challenge, right?

[00:16:52] Like if you define absolutely everything inside the database, then if they also have like, let's say a custom role. Actually, you can't really decide for them, should, you know, people with that custom role then automatically get access to this new feature.

[00:17:08] Like, it gets a little bit dicey, so like, we keep all of that inside the code.

[00:17:14] For the predefined roles, and then for the other roles, these are defined on the workspace. So they have, like, each user, which roles they have, which additional scopes they have. And then they also have custom roles. And then the custom roles can be referenced by the users that have roles assigned to them.

[00:17:37] So this is also an important thing is that like each user, even though in the interface, we only allow them to have one role right now, it's modeled in a way that they could have multiple additive roles in case we need that.

[00:17:49] Because we all know that like, you never want to start with a string. You always want to start with an array of strings or like an array of whatever. Because you will need it one day. So that's like, everything is array.

[00:18:04] Benedicte: I feel like this is a highlight from this one. Never, you know, you never want to start with like the one thing you want to start with an array of the thing, cause it's so much easier.

[00:18:13] Monica: Everytime Oh my God.

[00:18:17] Benedicte: I never thought about it in that way. Like that is a great way of putting it.

[00:18:20] Cause it's so much easier to then make it into multiple. And if it just stays as one, it's not really that big of an overhead.

[00:18:31] Monica: Totally. We built the product originally thinking that every publisher had one website. So you can imagine that as soon as we had to like change one of these core pieces to make it so they can have multiple, it was a lot of work.

[00:18:44] Benedicte: So you've learned this from experience?

[00:18:46] Monica: Oh, of course.

[00:18:50] Benedicte: Do you have any formal like data modeling education or has it all been trial and error?

[00:18:58] Monica: I mean, I like, I learned to code, like, from books as a kid. But I did like, take a minor in computer science in university. I didn't take the data structures and algorithms class, though, so I'm not really sure if I can say.

[00:19:13] But like, you know, I had to learn like Java and C. And of course you like, you learn an assembly, so, like, you learn kind of the fundamentals of like, how that all works I suppose.

[00:19:24] But no, I think there's a big difference when it comes to like building applications, especially those that have like end users versus like scripts and like the kind of stuff that you tend to build like for classes where it's like games or whatever.

[00:19:39] Like these are very different types of code. Like application code versus scripting and projects that you do for school.

[00:19:46] Benedicte: Yeah. I feel the same way. Cause I actually took both the algorithm and data structure and also a database class, I think. Cause I have a master in computer science.

[00:19:56] But as you said, like these things do not stick. Yeah. So fancy, but they do not stick when you don't use them in any real world applications.

[00:20:04] And also I feel like, especially in the database course, you know, it was all about kind of like that perfect database that I've never seen in production or talk to anyone about. Because it's you know, the normalization, yeah, where nothing should be repeated.

[00:20:22] And then you start like working on real data. You're like, "well, if we're going to be able to do this fast enough, we need to have, you know, this data repeated several places or, you know."

[00:20:31] So I feel like a lot of that, I mean, it's good probably to have as like a foundation, but I feel like data in production is a lot more different than in theory.

[00:20:43] And that's also why we wanted to do this podcast and hear about how it's actually, you know, being done and where the problems actually arise.

[00:20:52] Ola: So what data model has changed the most for you since you launched Affilimate?

[00:20:58] Monica: Hmm. I have to think about this.

[00:21:02] I mean, I know which data model has probably changed the least. And this is like our events data. So that's like consistent, you almost never need to touch it.

[00:21:10] But I don't know, I guess like as we were kind of talking about before, like our transaction data is maybe a model that just keeps getting bigger and bigger because we integrate with so many data sources and you want to kind of capture as much data as possible.

[00:21:27] But at the same time, you have to put it all into one format, which means there is like a trade off between being precise and being complete. So I would say that one is certainly like interesting because we have to handle all of the different transaction models out there and then somehow bring them into something that is like coherent on our end and allows us to like run the type of queries that we need to run in a fast way.

[00:21:53] So that one, it's like every day I have like new fields that I would be ready to add to it if it was not such like. You know, obviously like take some work to like add new columns, map the data, you know, re pull data, et cetera. So that one is getting more and more columns.

[00:22:13] Ola: I see. So if you could time travel back in time and change something about that data model to make it kind of change the decision or something, would that help?

[00:22:23] Monica: I mean, all I can say is that I wouldn't have put the data. Like first we started with having that data in Firebase, which was a bad idea because in Firebase and other document storages, you can't sum anything. And document based, like, a lot of times what you have to do is, like, duplicate the data and then sum things together, but this doesn't work.

[00:22:47] Like, it's not designed for transaction data. So, I think the biggest thing is that I would have used, like, a very, very boring, SQL database from the beginning. But obviously like that doesn't become a problem until you're like dealing with data of like a certain size. If you're dealing with small data, then it doesn't really matter because you can load everything in the client and do your addition there.

[00:23:10] But once you have to start loading, you know, like I would say for us, we have to start offloading as much of the labor as possible onto the databases and really choose purpose built databases for the specific tasks. And that's why we have so many databases.

[00:23:30] So, I would say that's maybe the biggest thing is like, I don't know. Storing the data in the right format in the right database from the beginning obviously would have saved a lot of time.

[00:23:41] Ola: But you didn't find that out before you started doing it.

[00:23:44] Monica: Well, we thought that our customers would be small and turns out they're not.

[00:23:50] Ola: That's the reason. Okay.

[00:23:53] Monica: Yeah. I mean, it's like, yeah. It's things were like for us, like performance optimizations and like thinking about like, okay, how will this look with a million rows?

[00:24:01] What about if we have a billion rows? Like, you know, it's just, you start to get to a size where those types of things matter. And of course, at the beginning, when you have like five users, it's like, it doesn't matter. But thankfully we have overcome most of that by now.

[00:24:19] Benedicte: So like, what are the most rows, if there are rows, that you have?

[00:24:27] Monica: I mean, I guess I'm not going to be too specific, but I would say like, I mean, we have billions and billions of rows of event data. Yeah. And it's all text data, which is crazy. So when you look at it, I mean, it's like. I mean, it's terabytes upon terabytes of text data. So there's a lot of it.

[00:24:49] Benedicte: Cause you were saying that transactions were maybe the ones that changed the most, and you said that the events was the one that changed the least. And that's the one you have the most of so.

[00:25:00] Monica: Yeah, it has like the fewest columns. But it has. We have the biggest volume of them because we also track impressions. So we track like page views and link clicks, but we also track impressions. So when someone views a link on a publisher website, we send, like batch together some events for that. And of course, like that means that anytime someone is scrolling, like we're going to be capturing data. So it's like, it's a lot.

[00:25:31] But this of course, like, it has benefits, but it's like, it is by far the most data that we have in our whole system is impression data. There's just tons of it. There's just so much impression data.

[00:25:45] Benedicte: I found that interesting because I haven't done any kind of larger, you know, like systems where I've been responsible for the. But we created Prune Your Follows where you can import all your, all the people you follow on Twitter to find people to unfollow.

[00:26:01] And I was just amazed at like how fast you kind of rack up the amount of rows.

[00:26:07] Monica: Yeah.

[00:26:08] Benedicte: And because we wanted it to be, you know, filterable, we ended up having it so that if two people import the same person, it's two rows because it made some of the search features easier.

[00:26:21] So I think we're already above like a million unique, like unique ones and like a million and at 1.6M or something like imported, rows or count rows. So and that was just, and that is still a small app. Like it's not like a, you know, huge enterprise thing. But it just like, yeah, the rows just adds up. Like it's.

[00:26:44] Monica: Especially if you are like, yeah, when you are importing or collecting data from a different data source, right?

[00:26:52] It's like different if it was, I don't know, I always make the comparison of like something like Teachable. Or it's like, okay, you need to build a course. How many modules could you possibly have in a course? Like, no one's gonna put a thousand modules in a course. Maybe they do, but even 1, 000 is a small number when it comes to databases, right?

[00:27:09] But then as soon as like, you're in your situation where you're like, I'm importing Twitter followers. This person managed to follow 50, 000 people or whatever is the upper limit. Then suddenly you're like, I have 50, 000 rows? And also when the user is like very small, but they just have like really disproportionate data.

[00:27:28] We have this sometimes where we have customers that have like, you know, their account itself on its own might be small, but then they have like these programmatically generated websites that have all of these pages. And it's like, oh my God, even Google is not crawling all of this because they don't care.

[00:27:49] So then you're like, have that mismatch between like data consumption and the actual like billable size, so to speak, of like the end user.

[00:27:57] Benedicte: So how many pages would they then programmatically create for a topic or for their website?

[00:28:03] Monica: I would say like the biggest that I've seen where, at least for now, we told them that we are not working with this type of site.

[00:28:13] Like number one, forums are a big one. So, like, if someone wants to like, track a forum, you can imagine. I mean, it's just all this user generated content. There's tons of new content every day. It really racks up, especially if they've been, had the forum has been around for a long time.

[00:28:30] But like we've had users where, like, you know, they're auto generating content, and they have millions of pages. And it's just like an incredible long tail of pages.

[00:28:42] Benedicte: A million pages? That is insane.

[00:28:44] Monica: No, millions. Millions.

[00:28:45] Benedicte: Millions! Oh, not even a million, like millions. That is insane!

[00:28:50] Monica: I mean, the thing is they're not like pre rendering all of these, right? It's not like a, You know what I mean?

[00:28:57] Like, it's all interlinked and auto generated. And like, you know, it's dynamic, so it's not like they havesome build task that creates a static website with a million pages. Obviously, like, this would be really slow. Also, just in terms of like, writing to disk, like, all of those million pages.

[00:29:16] Yeah, yeah, yeah.

[00:29:17] So, they don't have that. I mean, it's more dynamic.

[00:29:20] But yeah, those are some like, extreme cases.

[00:29:25] Benedicte: But I think we kind of, we're coming to a close. So I just want to say thank you so much for joining us today, Monica.

[00:29:33] I learned a lot, and I do hope the listener remembers. To always start with an array , and not just,

[00:29:41] Monica: That's the tagline.

[00:29:43] Benedicte: That's the tagline. Always start with an array.

[00:29:45] Don't just do a string. Do an array of string. Don't just do, I guess a number, an array.

[00:29:50] Monica: An object.

[00:29:51] Benedicte: An object. Just do an array. Do an array. If you never need it to be an array, it's still worth it.

[00:29:57] And we want to ask you where people can go to find out more about you and Affilimate.

[00:30:03] Monica: Yeah, sure.

[00:30:04] I'm on Twitter. It's twitter.com/monicalent And then Affilimate is just at affilimate.com We are also on LinkedIn. So if you want, you can also follow our page on LinkedIn.

[00:30:22] But yeah. Twitter is, I guess, where I log in the most. But these days I have the LinkedIn app on my phone.

[00:30:27] Benedicte: And you can also check out the description in your podcast player for these links. Until next week, bye bye.

[00:30:37] Ola: Bye bye.

[00:30:38] Monica: Bye.