User:Razzi/First logical volume resizing
puppet $ sshrc an-test-coord1001.eqiad.wmnet Linux an-test-coord1001 4.19.0-16-amd64 #1 SMP Debian 4.19.181-1 (2021-03-19) x86_64 Debian GNU/Linux 10 (buster) _ __ _ _ _ _ _ | |/ / | | (_) | | | | | | | ' / ___ _ __| |__ ___ _ __ _ _______ __| | | |__ ___ ___| |_ | < / _ \ '__| '_ \ / _ \ '__| |_ / _ \/ _` | | '_ \ / _ \/ __| __| | . \ __/ | | |_) | __/ | | |/ / __/ (_| | | | | | (_) \__ \ |_ |_|\_\___|_| |_.__/ \___|_| |_/___\___|\__,_| |_| |_|\___/|___/\__| This host is capable of Kerberos authentication in the WIKIMEDIA realm. For more info: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Kerberos/UserGuide an-test-coord1001 is a Analytics Cluster host running various Hadoop services (Hive, Camus, Oozie, ..) and maintenance scripts (analytics_test_cluster::coordinator) Debian GNU/Linux 10 auto-installed on Fri Apr 2 14:23:58 UTC 2021. Last login: Mon Jan 24 16:57:47 2022 from 2620:0:863:1:198 You do not have a valid Kerberos ticket in the credential cache, remember to kinit. welcome razzi@an-test-coord1001:~$ ls apache-atlas-2.2.0-sources.tar cookiejar.txt pidinfo.txt apache-atlas-sources-2.2.0 index.html run_atlas razzi@an-test-coord1001:~$ df -h df: /mnt/hdfs: Input/output error Filesystem Size Used Avail Use% Mounted on udev 63G 0 63G 0% /dev tmpfs 13G 1.4G 12G 11% /run /dev/mapper/vg0-root 73G 50G 20G 72% / tmpfs 63G 0 63G 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 63G 0 63G 0% /sys/fs/cgroup /dev/mapper/vg0-srv 365G 29G 318G 9% /srv tmpfs 13G 0 13G 0% /run/user/0 tmpfs 13G 0 13G 0% /run/user/118 tmpfs 13G 0 13G 0% /run/user/32741 tmpfs 13G 0 13G 0% /run/user/2543 tmpfs 13G 0 13G 0% /run/user/26051 razzi@an-test-coord1001:~$ man lvreduce razzi@an-test-coord1001:~$ sudo lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert root vg0 -wi-ao---- 74.50g srv vg0 -wi-ao---- <371.27g swap vg0 -wi-ao---- 976.00m razzi@an-test-coord1001:~$ lvreduce srv -L -2 bash: lvreduce: command not found razzi@an-test-coord1001:~$ sudo lvreduce srv -L -2 Please specify a logical volume path. Run `lvreduce --help' for more information. razzi@an-test-coord1001:~$ sudo lvreduce /dev/mapper/vg0-srv -L -2G WARNING: Reducing active and open logical volume to <369.27 GiB. THIS MAY DESTROY YOUR DATA (filesystem etc.) Do you really want to reduce vg0/srv? [y/n]: y Size of logical volume vg0/srv changed from <371.27 GiB (95044 extents) to <369.27 GiB (94532 extents). Logical volume vg0/srv successfully resized. razzi@an-test-coord1001:~$ sudo lvreduce /dev/mapper/vg0-root -L +2G Size may not be positive. Invalid argument for --size: +2G Error during parsing of command line. razzi@an-test-coord1001:~$ sudo lvextend /dev/mapper/vg0-root -L+2G Size of logical volume vg0/root changed from 74.50 GiB (19073 extents) to 76.50 GiB (19585 extents). Logical volume vg0/root successfully resized. razzi@an-test-coord1001:~$ resize2fs /dev/mapper/vg0-root bash: resize2fs: command not found razzi@an-test-coord1001:~$ sudo resize2fs /dev/mapper/vg0-root resize2fs 1.44.5 (15-Dec-2018) Filesystem at /dev/mapper/vg0-root is mounted on /; on-line resizing required old_desc_blocks = 10, new_desc_blocks = 10 The filesystem on /dev/mapper/vg0-root is now 20055040 (4k) blocks long. razzi@an-test-coord1001:~$ df -h df: /mnt/hdfs: Input/output error Filesystem Size Used Avail Use% Mounted on udev 63G 0 63G 0% /dev tmpfs 13G 1.4G 12G 11% /run /dev/mapper/vg0-root 75G 50G 22G 70% / tmpfs 63G 0 63G 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 63G 0 63G 0% /sys/fs/cgroup /dev/mapper/vg0-srv 365G 29G 318G 9% /srv tmpfs 13G 0 13G 0% /run/user/0 tmpfs 13G 0 13G 0% /run/user/118 tmpfs 13G 0 13G 0% /run/user/32741 tmpfs 13G 0 13G 0% /run/user/2543 tmpfs 13G 0 13G 0% /run/user/26051
I did the above in the wrong order; we were fortunately able to create a backup of mariadb from the corrupted volume and restore that to srv. Some other folders were manually recreated and some were created from a puppet run. The puppet run errored however, with a systemd issue:
Message from syslogd@an-test-coord1001 at Jan 24 20:57:01 ... systemd[1]: Caught <ABRT>, dumped core as pid 15168. Broadcast message from systemd-journald@an-test-coord1001 (Mon 2022-01-24 20:57:01 UTC): systemd[1]: Caught <ABRT>, dumped core as pid 15168. Message from syslogd@an-test-coord1001 at Jan 24 20:57:01 ... systemd[1]: Freezing execution. Broadcast message from systemd-journald@an-test-coord1001 (Mon 2022-01-24 20:57:01 UTC): systemd[1]: Freezing execution. razzi@an-test-coord1001:~$