Using Swagger UI For Local Development

When working on the swagger documentation, several different tools can be used.

The ‘official’ editor developed by the Swagger community is Swagger Editor, live preview: https://editor.swagger.io/, I have found it quite quick and easy to use, but for larger project it gets cumbersome. Additionally it doesn’t support ability to have multiple files that will reference each other.

This post is a quick demonstration of the workflow I’m using when working on the large, multi-file swagger documentations. It’s certainly not ideal but it helps me to get the job done.

Here it is:

Edit swagger.yaml

This file can be edited in any editor of choice, the yaml format is widely supported. The documentation can be split up into multiple files that reference each other. I use this functionality to extract examples and other JSON documents. Referencing other files is explained quite well in this post How to split a Swagger spec into smaller files

Publish swagger.yaml locally

In order to view swagger.yaml inside Swagger UI the file can be either referenced by a regular file system reference, or it can be fetched via HTTP.

I use local HTTP server to expose swagger.yaml (with all other, referenced files)

Here I’m using http-server from npm but you can use any HTTP server, make sure to disable caching and CORS

Run Swagger UI

The last component, Swagger UI can be started like this:

After viewing http://localhost:1111 Swagger UI will display the generated documentation

Now, every time you make a change swagger.yaml or any file it’s enough to refresh web page to see the changes (or validation errors if there are some)

Summary

This method is more involved that alternatives like Swagger Editor or simply running Swagger UI with file referenced by regular file system reference (possibly outside Docker), but overcomes 2 major limitations of those tools, with it you’ll be able to:

  • Not pollute your environment with different tools when Docker container will work
  • Ability to edit swagger documentation that consist of multiple files (either JSON or YAML)
  • Quick feedback loop, simply refresh web page to see the changes in swagger docs

Scala Patterns To Avoid: Implicit Arguments With Default Values

There is a tendency for the Scala projects to prefer more explicit programming style. The biggest aspect of that is in my opinion the type system of the Scala language, programmers often start writing their functions by defining types of the arguments and type of the result, only to write the body of the function as last step. That’s also because we have the Scala compiler to help us.

I recently stumbled on a snippet that contradicts this rule and can be a source of hard to spot bugs for the person unfamiliar with the code.

The problem happens when you try to mix both features of the Scala language that make sense if used on their own (although I’ll argue with that a little in a moment), but when used together they create a very serious problem.

The features (as you probably guessed) are:

  • Implicit arguments
  • Default function arguments values

Implicit arguments

Implicit arguments when used in separation make a lot of sense, they allow for context-like arguments to be passed through the whole call stack. Furthermore they are the major building block for more advanced features like typeclasses.
Many major libraries or projects would not exist without it, but because this is a very powerful feature, it’s use should be limited.

Default function arguments

In my option this feature should be avoided almost everywhere – they make it very hard to use functions as arguments and pass function around.
It’s better to use either currying (multiple parameter lists) or just define function few times taking different set of parameters in each case.

I think it makes sense to use default arguments in 2 cases:

  • Backwards compatibility – you have an existing case class and want do add another attribute to it without changing your code or underlying database structures
  • Complicated, builder-like constructors – you have constructor that takes a ton of arguments, and users would like to configure only a limited set each time, using defaults for everything else

The problematic code

Here’s a snippet of the code I stumbled on:

Here’s what I think is wrong here:

  • Caller is not aware that updateCampaign actually takes any argument unless he/she reads the source or looks up documentation (if there’s any).
    You can perfectly well write code that looks like this:

And you will not see anything wrong – the compiler won’t complain. Unless the caller is aware there’s another parameter expected, there’s no straightforward way to learn it.

  • If someone overrides updateCampaign function this information will be lost forever, all calls to the overridden version will use the default argument.

The above snippet contradicts the explicitness rule, also Scala compiler will not help you spotting a bug.

Of course the code will still run – the only problem is that every update to the campaign will be attributed to User.Default which is not what we are expecting – but there’s no way to express that intend when using default arguments.

Fixing the code

To fix it, simply remove the default value for the user argument:

After that change you will get the compilation error forcing you to fix the issue.

Scala compiler is now able to detect this issue right away, you don’t have to remember which functions take which arguments because you can leverage the compiler.

Summary

The outlined scenario is one of many examples where a programmer can use Scala compiler to his/her benefit. A simple change to the code will result in a compiler pointing out the problem right away, forcing you to address it before code is released.

It’s also one of the many logic-related mistakes that are very hard to spot in testing – the “audit logging” in this case is a side effect to the regular responsibility of the code.

Flyway Database Migrations – Best Practices

This post is attempts cover some of best practices in using Flyway, the database migration tool, that I have learned or established after using it in few projects. It’s aimed at users already familiar with Flyway or any other database migration tools, so I’ll try to skip over the very basics.

I’m focusing on Flyway here, but I think the best practices will apply to majority of other database migration tools.

Transactional Migrations

Each individual database migration is wrapped by Flyway inside a transaction. This means that:

  • You shouldn’t use transactions explicitly in your plain SQL migrations
  • If you are running multiple transactions at once (for example updating from version 1 to 4), this means that some it will stop running after first failure – some migrations will work, some won’t.

Dealing With Rollbacks

Flyway doesn’t support rollbacks explicitly, instead you should always create new migration that should revert recently introduced changes. This is the same patter used by git revert $COMMIT.

Rollbacks During Development

During development those rules should be more relaxed. When you are working on creating correct database models or just migrations, you will often have a need to reset database at the specific state and re-run the migration. I have always found this a little annoying and here’s the process I’ve been following:

(Please note that this is a very bad idea to do this on the production database, but during development that’s in fact a good idea)

  • After I realize that I need to modify the last migration I have executed
  • Manually revert database changes
  • Delete last entry from schema_version
  • Run flywayInfo to confirm that Flyway doesn’t see my last migration
  • Run flywayMigrate to ‘re-run’ migration

I have found this method to work well in practice, especially when each individual migration is small and the iteration cycle is small. The only requirement is that the database migration I’m editing hasn’t been applied anywhere else outside of my local environment.

Use Baseline At The Start

Baseline command is used to introduce a database migration tool to the existing project, here’s how you should use it:

  • Create a database dump as SQL file
  • Add this file to Flyway by running flywayBaseline introducing this file as V1__Baseline.sql

What many people fail to do is to manually inspect the database dump to eliminate unnecessary or unwanted data. You should cleanup the database dump so it will contain only “database schemas” and “reference information”.

Performance Tips

In cases when you are adding a large piece of reference data – tables with thousands or hundreds of thousands of rows, it’s good idea performance-wise to follow following schematics:

Creating indexes after loading in the data speeds up the process, because the database doesn’t need to keep the index up to date as you add rows, it will only perform a single index creation operation, the bigger your table is, the bigger the impact.

Missing Migrations Warning

It’s possible to programmatically inspect schema_version (You can also use Flyway API but this requires adding dependency to your project) table and issue warning in the following cases:

  • You migrations that were not executed
  • You have migrations that resulted in errors
  • You have more migrations that were performed that application knows about
  • Etc…

Missing Migrations Error

This idea builds on the previous one, instead of issuing warning you can selectively issue either warnings or errors or even exit your application when it detects serious problems with the state of the database.

Setup Database Users

Flyway will display information about which user performed the migration, each user/operator should have it’s own account in the database so this information will be persisted. Avoid using generic users or same user that your application uses for accessing database.

Don’t Create Users With Migrations

User and permission management shouldn’t be handled by database migrations. Here’s the reason behind it:

If you run your application in new or different environment you might need to create different users or setup different permissions (for example IP addresses) – I have found it easier not to include users as part of migration procedure.

Editing Previously Executed Migration

Let’s say you have found a bug in the migration that’s already been executed on the production database. You have 2 options here:

Create new migration that will fix the bug

This is the ideal scenario, but sometimes you would like to take the shortcut

Advanced Users can take the shortcut

There is a limited set of cases when the shortcut can be taken, I don’t have the list, so use your own judgement 🙂

Here are the rough steps:

  • Edit migration file in place
  • Manually perform the diff migration between what’s in the database and your new migration file (*)
  • Run flywayInfo, you should get an exception complaining about checksum mismatch
  • Edit row in schema_version table updating checksum from existing value to the new one

(*) In order for this method to work it should be possible to create that diff and it should be relatively simple to perform to ensure you don’t introduce any issues

Summary

Flyway (or SQL database migration tools in general) is a tool that in my opinion is a must have for any serious project.

I have shown some of the patterns and best practices that I have worked out after using if in multiple projects.