User:Razzi/Superset 1.3.1 upgrade recap

From Wikitech

I upgrade superset to 1.3.1 and ran into a couple snags upon the way. This is the raw information dump; I'll try to collect the takeaways and update the actual superset administration article https://wikitech.wikimedia.org/wiki/Analytics/Systems/Superset#Upgrading

Here is the original session, where I hit the sql error at the end which caused about 30 minutes of downtime. 

Last login: Wed Nov  3 09:58:50 on ttys006
>*)))>< <*)))>< "how is your posture?"
~ 09:59 $ sshrc an-tool1010.e
Linux an-tool1010 4.19.0-14-amd64 #1 SMP Debian 4.19.171-2 (2021-01-30) x86_64
Debian GNU/Linux 10 (buster)

  _  __         _               _             _   _               _   
 | |/ /        | |             (_)           | | | |             | |  
 | ' / ___ _ __| |__   ___ _ __ _ _______  __| | | |__   ___  ___| |_ 
 |  < / _ \ '__| '_ \ / _ \ '__| |_  / _ \/ _` | | '_ \ / _ \/ __| __|
 | . \  __/ |  | |_) |  __/ |  | |/ /  __/ (_| | | | | | (_) \__ \ |_ 
 |_|\_\___|_|  |_.__/ \___|_|  |_/___\___|\__,_| |_| |_|\___/|___/\__|


This host is capable of Kerberos authentication in the WIKIMEDIA realm.

For more info: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Kerberos/UserGuide

an-tool1010 is a Analytics Superset web interface (analytics_cluster::ui::superset)
Debian GNU/Linux 10 auto-installed on Thu Dec 10 12:36:53 UTC 2020.
Last login: Thu Oct 28 15:55:15 2021 from 2620:0:863:1:198

You do not have a valid Kerberos ticket in the credential cache, remember to kinit.
welcome
razzi@an-tool1010:~$ source /srv/deployment/analytics/superset/venv/bin/activate
(venv) razzi@an-tool1010:~$ export PYTHONPATH=/etc/superset
(venv) razzi@an-tool1010:~$ superset db upgrade
Found but failed to import local superset_config
Traceback (most recent call last):
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/superset/config.py", line 1267, in <module>
    import superset_config  # pylint: disable=import-error
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 724, in exec_module
  File "<frozen importlib._bootstrap_external>", line 859, in get_code
  File "<frozen importlib._bootstrap_external>", line 916, in get_data
PermissionError: [Errno 13] Permission denied: '/etc/superset/superset_config.py'
Traceback (most recent call last):
  File "/srv/deployment/analytics/superset/venv/bin/superset", line 5, in <module>
    from superset.cli import superset
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/superset/cli.py", line 35, in <module>
    from superset import app, appbuilder, config, security_manager
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/superset/config.py", line 1267, in <module>
    import superset_config  # pylint: disable=import-error
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 724, in exec_module
  File "<frozen importlib._bootstrap_external>", line 859, in get_code
  File "<frozen importlib._bootstrap_external>", line 916, in get_data
PermissionError: [Errno 13] Permission denied: '/etc/superset/superset_config.py'
(venv) razzi@an-tool1010:~$ sudo su superset
superset@an-tool1010:/home/razzi$ source /srv/deployment/analytics/superset/venv/bin/activate
(venv) superset@an-tool1010:/home/razzi$ export PYTHONPATH=/etc/superset
(venv) superset@an-tool1010:/home/razzi$ superset db upgrade
Loaded your LOCAL configuration at [/etc/superset/superset_config.py]
logging was configured successfully
2021-11-03 17:00:14,236:INFO:superset.utils.logging_configurator:logging was configured successfully
2021-11-03 17:00:14,245:INFO:root:Configured event logger of type <class 'superset.utils.log.DBEventLogger'>
/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/flask_caching/__init__.py:202: UserWarning: Flask-Caching: CACHE_TYPE is set to null, caching is effectively disabled.
  "Flask-Caching: CACHE_TYPE is set to null, "
No PIL installation found
2021-11-03 17:00:14,507:INFO:superset.utils.screenshots:No PIL installation found
INFO  [alembic.runtime.migration] Context impl MySQLImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade 67da9ef1ef9c -> 989bbe479899, rename_filter_configuration_in_dashboard_metadata.py
Updated 0 native filter configurations.
INFO  [alembic.runtime.migration] Running upgrade 989bbe479899 -> 301362411006, add_execution_id_to_report_execution_log_model.py
INFO  [alembic.runtime.migration] Running upgrade 301362411006 -> 134cea61c5e7, remove dataset health check message
INFO  [alembic.runtime.migration] Running upgrade 134cea61c5e7 -> 085f06488938, Country map use lowercase country name
INFO  [alembic.runtime.migration] Running upgrade 085f06488938 -> fc3a3a8ff221, migrate filter sets to new format
Updated 0 filter sets with 0 filters.
INFO  [alembic.runtime.migration] Running upgrade fc3a3a8ff221 -> 19e978e1b9c3, add_report_format_to_report_schedule_model.py
INFO  [alembic.runtime.migration] Running upgrade 19e978e1b9c3 -> d416d0d715cc, add_limiting_factor_column_to_query_model.py
Traceback (most recent call last):
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
    cursor, statement, parameters, context
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
    cursor.execute(statement, parameters)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/MySQLdb/cursors.py", line 206, in execute
    res = self._query(query)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/MySQLdb/cursors.py", line 312, in _query
    db.query(q)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/MySQLdb/connections.py", line 224, in query
    _mysql.connection.query(self, query)
MySQLdb._exceptions.OperationalError: (1118, 'Row size too large. The maximum row size for the used table type, not counting BLOBs, is 8126. This includes storage overhead, check the manual. You have to change some columns to TEXT or BLOBs')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/srv/deployment/analytics/superset/venv/bin/superset", line 8, in <module>
    sys.exit(superset())
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/flask/cli.py", line 586, in main
    return super(FlaskGroup, self).main(*args, **kwargs)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/flask/cli.py", line 426, in decorator
    return __ctx.invoke(f, *args, **kwargs)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/flask_migrate/cli.py", line 149, in upgrade
    _upgrade(directory, revision, sql, tag, x_arg)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/flask_migrate/__init__.py", line 98, in wrapped
    f(*args, **kwargs)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/flask_migrate/__init__.py", line 185, in upgrade
    command.upgrade(config, revision, sql=sql, tag=tag)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/alembic/command.py", line 320, in upgrade
    script.run_env()
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/alembic/script/base.py", line 563, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/alembic/util/pyfiles.py", line 92, in load_python_file
    module = load_module_py(module_id, path)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/alembic/util/pyfiles.py", line 108, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/superset/migrations/env.py", line 124, in <module>
    run_migrations_online()
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/superset/migrations/env.py", line 116, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/alembic/runtime/environment.py", line 851, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/alembic/runtime/migration.py", line 612, in run_migrations
    step.migration_fn(**kw)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/superset/migrations/versions/d416d0d715cc_add_limiting_factor_column_to_query_.py", line 36, in upgrade
    sa.Column("limiting_factor", sa.VARCHAR(255), server_default="UNKNOWN",)
  File "/usr/lib/python3.7/contextlib.py", line 119, in __exit__
    next(self.gen)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/alembic/operations/base.py", line 374, in batch_alter_table
    impl.flush()
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/alembic/operations/batch.py", line 107, in flush
    fn(*arg, **kw)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/alembic/ddl/impl.py", line 324, in add_column
    self._exec(base.AddColumn(table_name, column, schema=schema))
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/alembic/ddl/impl.py", line 197, in _exec
    return conn.execute(construct, multiparams)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
    return meth(self, multiparams, params)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/sqlalchemy/sql/ddl.py", line 72, in _execute_on_connection
    return connection._execute_ddl(self, multiparams, params)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1073, in _execute_ddl
    compiled,
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1317, in _execute_context
    e, statement, parameters, cursor, context
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1511, in _handle_dbapi_exception
    sqlalchemy_exception, with_traceback=exc_info[2], from_=e
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
    cursor, statement, parameters, context
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
    cursor.execute(statement, parameters)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/MySQLdb/cursors.py", line 206, in execute
    res = self._query(query)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/MySQLdb/cursors.py", line 312, in _query
    db.query(q)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/MySQLdb/connections.py", line 224, in query
    _mysql.connection.query(self, query)
sqlalchemy.exc.OperationalError: (MySQLdb._exceptions.OperationalError) (1118, 'Row size too large. The maximum row size for the used table type, not counting BLOBs, is 8126. This includes storage overhead, check the manual. You have to change some columns to TEXT or BLOBs')
[SQL: ALTER TABLE query ADD COLUMN limiting_factor VARCHAR(255) DEFAULT 'UNKNOWN']
(Background on this error at: http://sqlalche.me/e/13/e3q8)
(venv) superset@an-tool1010:/home/razzi$ sudo systemctl stop superset

We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:

    #1) Respect the privacy of others.
    #2) Think before you type.
    #3) With great power comes great responsibility.

[sudo] password for superset: 
# whoops! Tried to run an sre command as superset user
(venv) superset@an-tool1010:/home/razzi$ exit
(venv) razzi@an-tool1010:~$ sudo systemctl stop superset
(venv) razzi@an-tool1010:~$ exit
Connection to an-tool1010.eqiad.wmnet closed.


(venv) superset@an-tool1010:/home/razzi$ superset db upgrade
Loaded your LOCAL configuration at [/etc/superset/superset_config.py]
logging was configured successfully
2021-11-03 17:13:09,532:INFO:superset.utils.logging_configurator:logging was configured successfully
2021-11-03 17:13:09,541:INFO:root:Configured event logger of type <class 'superset.utils.log.DBEventLogger'>
/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/flask_caching/__init__.py:202: UserWarning: Flask-Caching: CACHE_TYPE is set to null, caching is effectively disabled.
  "Flask-Caching: CACHE_TYPE is set to null, "
No PIL installation found
2021-11-03 17:13:09,798:INFO:superset.utils.screenshots:No PIL installation found
INFO  [alembic.runtime.migration] Context impl MySQLImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade 19e978e1b9c3 -> d416d0d715cc, add_limiting_factor_column_to_query_model.py
Traceback (most recent call last):
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
    cursor, statement, parameters, context
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
    cursor.execute(statement, parameters)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/MySQLdb/cursors.py", line 206, in execute
    res = self._query(query)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/MySQLdb/cursors.py", line 312, in _query
    db.query(q)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/MySQLdb/connections.py", line 224, in query
    _mysql.connection.query(self, query)
MySQLdb._exceptions.OperationalError: (1118, 'Row size too large. The maximum row size for the used table type, not counting BLOBs, is 8126. This includes storage overhead, check the manual. You have to change some columns to TEXT or BLOBs')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/srv/deployment/analytics/superset/venv/bin/superset", line 8, in <module>
    sys.exit(superset())
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/flask/cli.py", line 586, in main
    return super(FlaskGroup, self).main(*args, **kwargs)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/flask/cli.py", line 426, in decorator
    return __ctx.invoke(f, *args, **kwargs)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/flask_migrate/cli.py", line 149, in upgrade
    _upgrade(directory, revision, sql, tag, x_arg)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/flask_migrate/__init__.py", line 98, in wrapped
    f(*args, **kwargs)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/flask_migrate/__init__.py", line 185, in upgrade
    command.upgrade(config, revision, sql=sql, tag=tag)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/alembic/command.py", line 320, in upgrade
    script.run_env()
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/alembic/script/base.py", line 563, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/alembic/util/pyfiles.py", line 92, in load_python_file
    module = load_module_py(module_id, path)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/alembic/util/pyfiles.py", line 108, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/superset/migrations/env.py", line 124, in <module>
    run_migrations_online()
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/superset/migrations/env.py", line 116, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/alembic/runtime/environment.py", line 851, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/alembic/runtime/migration.py", line 612, in run_migrations
    step.migration_fn(**kw)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/superset/migrations/versions/d416d0d715cc_add_limiting_factor_column_to_query_.py", line 36, in upgrade
    sa.Column("limiting_factor", sa.VARCHAR(255), server_default="UNKNOWN",)
  File "/usr/lib/python3.7/contextlib.py", line 119, in __exit__
    next(self.gen)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/alembic/operations/base.py", line 374, in batch_alter_table
    impl.flush()
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/alembic/operations/batch.py", line 107, in flush
    fn(*arg, **kw)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/alembic/ddl/impl.py", line 324, in add_column
    self._exec(base.AddColumn(table_name, column, schema=schema))
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/alembic/ddl/impl.py", line 197, in _exec
    return conn.execute(construct, multiparams)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
    return meth(self, multiparams, params)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/sqlalchemy/sql/ddl.py", line 72, in _execute_on_connection
    return connection._execute_ddl(self, multiparams, params)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1073, in _execute_ddl
    compiled,
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1317, in _execute_context
    e, statement, parameters, cursor, context
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1511, in _handle_dbapi_exception
    sqlalchemy_exception, with_traceback=exc_info[2], from_=e
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
    cursor, statement, parameters, context
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
    cursor.execute(statement, parameters)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/MySQLdb/cursors.py", line 206, in execute
    res = self._query(query)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/MySQLdb/cursors.py", line 312, in _query
    db.query(q)
  File "/srv/deployment/analytics/superset/venv/lib/python3.7/site-packages/MySQLdb/connections.py", line 224, in query
    _mysql.connection.query(self, query)
sqlalchemy.exc.OperationalError: (MySQLdb._exceptions.OperationalError) (1118, 'Row size too large. The maximum row size for the used table type, not counting BLOBs, is 8126. This includes storage overhead, check the manual. You have to change some columns to TEXT or BLOBs')
[SQL: ALTER TABLE query ADD COLUMN limiting_factor VARCHAR(255) DEFAULT 'UNKNOWN']
(Background on this error at: http://sqlalche.me/e/13/e3q8)

We tried updating the table manually, which may have caused the migrations to get out of sync with the database, but we got the same error: 

MariaDB [superset_production]> ALTER TABLE query ADD COLUMN limiting_factor VARCHAR(255) DEFAULT 'UNKNOWN';
ERROR 1118 (42000): Row size too large. The maximum row size for the used table type, not counting BLOBs, is 8126. This includes storage overhead, check the manual. You have to change some columns to TEXT or BLOBs

Dropping superset_production and recreating from snapshot worked, though we may have lost a couple edits between the time that the database was dumped and the ui crashed. 

razzi@an-coord1001:~$ cd /srv/
razzi@an-coord1001:/srv$ ls
backup            sqldata
deployment        superset_production_1631721039.sql
event-schemas     superset_production_1634597517.sql
lost+found        superset_production_1635958642.sql
mediawiki-config  tmp
presto
razzi@an-coord1001:/srv$ sudo mysql
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 134501851
Server version: 10.4.18-MariaDB-log MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> drop database superset_production;
Query OK, 55 rows affected (0.162 sec)

MariaDB [(none)]> create database superset_production DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
Query OK, 1 row affected (0.001 sec)

MariaDB [(none)]> Bye
razzi@an-coord1001:/srv$ sudo mysql superset_production < s
sqldata/                            superset_production_1634597517.sql
superset_production_1631721039.sql  superset_production_1635958642.sql
razzi@an-coord1001:/srv$ sudo mysql superset_production < superset_production_1635958642.sql
Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=User:Razzi/Superset_1.3.1_upgrade_recap&oldid=1931326"