Author Archives: olof

Convert MyISAM tables to InnoDB

Did you know that InnoDB replaces MyISAM as the default storage engine since MySQL 5.5.5? For this reason you may want to convert most (if not all) of your current MyISAM tables to InnoDB.

WordPress does for example make use of the default storage engine. That means old WordPress installations will make use of MyISAM if you don’t update them to use InnoDB instead.

SQL to get the job done

This is the SQL to convert a single table:

ALTER TABLE `databasename`.`tablename` ENGINE=InnoDB;

You can generate this SQL for all tables in a certain set of databases by using the following query:

SELECT CONCAT('ALTER TABLE `', table_schema, '`.`', TABLE_NAME, '` ENGINE=InnoDB;') AS query
FROM information_schema.TABLES
WHERE table_schema IN ('db1', 'db2');

For a list of the tables that use MyISAM run the following query:

SELECT table_schema, TABLE_NAME
FROM information_schema.TABLES
WHERE engine = 'MyISAM';

The above query will show you tables in the databases “information_schema” and “mysql” and you may not care to see those since they are system databases that should not be tampered with. This query excludes them from the results:

SELECT table_schema, TABLE_NAME
FROM information_schema.TABLES
WHERE engine = 'MyISAM' AND table_schema NOT IN('information_schema', 'mysql');

And why not combine it with the query generator while we are at it:

SELECT table_schema, TABLE_NAME, CONCAT('ALTER TABLE `', table_schema, '`.`', TABLE_NAME, '` ENGINE=InnoDB;') AS query
FROM information_schema.TABLES
WHERE engine = 'MyISAM' AND table_schema NOT IN('information_schema', 'mysql');

Backups for your linux server

Leave a reply

In this guide I describe how to setup backups for your Linux server. We will backup both locally and remotely for extra security. We will backup…

MySQL-databases using automysqlbackup.
Normal files using rdiff-backup.
Special large folders per weekday using rsync.
To a remote server using rsync.

Setting up the folder structure

We start out with creating a folder structure like this:

mkdir /backup
chmod 755 /backup
mkdir /backup/local
mkdir /backup/local/mysql
mkdir /backup/local/files

The new root-level folder “backup” is the storage location we will use for any kind of backups. The folder “local” inside it is where we will save the backup-files for that computer itself. Later on we will add sibling folders that contain backups from other computers.

Note that the folder “/backup” not necessarily need to be a folder on your main storage device. It could for example be a mount point or a symlink to a folder on another storage device. This makes a lot of sense if you main storage device is an SSD. Do not backup to an SSD.

Backup MySQL-databases

For MySQL-databases we will use the tool automysqlbackup. This tool is in the Ubuntu repositories. The versions are Ubuntu 12.04 with automysqlbackup 2.5 right now when I write this.

apt-get install automysqlbackup

We need to make two changes in the config file “/etc/default/automysqlbackup”:

DBNAMES=`mysql --defaults-file=/etc/mysql/debian.cnf --execute="SHOW DATABASES" | awk '{print $1}' | grep -Ev '^phpmyadmin|test|information_schema|performance_schema|mysql|Database$' | tr \\\r\\\n ,\ `
 
...
 
BACKUPDIR="/backup/local/mysql"

The grep part in DBNAMES is to exclude databases. You may want to add additional databases there that you don’t want to backup. You should now run it once to make sure it works:

automysqlbackup

Look through the contents in “/backup/local/mysql” to verify it looks the way you want it to.

Question: When will these backups run?
Answer: When installing automysqlbackup the file “/etc/cron.daily/automysqlbackup” was created. Check the content of “/etc/crontab” to see when daily cron runs. In my case it looks like this:

# m h dom mon dow user    command
17 *    * * *    root    cd / && run-parts --report /etc/cron.hourly
25 6    * * *    root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
47 6    * * 7    root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
52 6    1 * *    root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )

So it seems daily cron-tasks run at 06:25 on my system (Ubuntu 12.04).

Backup normal files

For backing up normal files we are going to use a tool called rdiff-backup that creates incremental backups.

apt-get install rdiff-backup

Question: How often should I backup?
Answer: I personally enjoy backing up once a day.

Question: How long should I save the backups?
Answer: I personally enjoy saving the backups for three months.

Question: What files should I backup?
Answer: A good start is “/home”, “/etc” and “/root”. You will however often need to setup some special exclude and include rules.

With this in mind I should create the file “/etc/cron.daily/backupfiles” with the following content:

#!/bin/bash
 
rdiff-backup \
--include /home \
--include /etc \
--include /root \
--exclude / \
/ /backup/local/files
 
rdiff-backup --force --remove-older-than 3M /backup/local/files

Ensure that file has the right permissions:

sudo chmod 755 /etc/cron.daily/backupfiles

Say however that I would like to ignore all home-files for the user “derp” and only want the folders “a” and “b” from the user “herp”.

The file would then look like this instead:

#!/bin/bash
 
rdiff-backup \
--include /home/herp/b \
--include /home/herp/a \
--exclude /home/herp \
--exclude /home/derp \
--include /home \
--include /etc \
--include /root \
--exclude / \
/ /backup/local/files
 
rdiff-backup --force --remove-older-than 3M /backup/local/files

It is important to consider that:

Multiple include and exclude options take precedence in the order they are given.
- http://www.nongnu.org/rdiff-backup/examples.html#exclude

I prefer looking at the include/exclude order like this:

The deeper the directory paths – the higher up in that cron-file they should be written.
We can visualize the end result by imagining we apply the includes/excludes from bottom up.

Backup normal files using crontab

I actually prefer having more fine grained control over when backups are run. For this we will use crontab. Start out with creating a folder for your cron-scripts:

mkdir /root/cron

And then create the file “/root/cron/backupfiles.sh” instead of the file “/etc/cron.daily/backupfiles”. Next we add a crontab entry as root:

crontab -e

If we want it to run the same time as automysqlbackup (06:25) the crontab entry would look like this:

#m      h       dom     mon     dow     command
25      6       *       *       *       /root/cron/backupfiles.sh

Backup special large folders per weekday

To be more precise I run a few minecraft servers. The map files are huge and change often because they are compressed natively. If we took incremental backups (rdiff-backup) those increments would be very large due to the compression. I prefer creating this setup instead:

mkdir /backup/local/server
mkdir /backup/local/server/1
mkdir /backup/local/server/2
mkdir /backup/local/server/3
mkdir /backup/local/server/4
mkdir /backup/local/server/5
mkdir /backup/local/server/6
mkdir /backup/local/server/7

Monday is “1″ and Sunday is “7″. Next we create the script “/root/cron/backupserver.sh”.

#!/bin/bash
rsync -avz --delete /home/mc/files/server/ /backup/local/server/$(date '+%u')

Note that after the first week the old backups will start getting overwritten.
Lets add it to the root crontab as well:

#m      h       dom     mon     dow     command
25      6       *       *       *       /root/cron/backupfiles.sh
35      6       *       *       *       /root/cron/backupserver.sh

Backup to a remote server

For extra security it makes sense to backup to a remote computer as well. Lets call the local computer “torkel” and the remote computer “berit”. Let’s also assume entries for those are added to the file “/etc/hosts” so they resolve into actual ip-addresses.

For this we could simply rsync the folder “torkel:/backup/local” to “berit:/backup/torkel” but in our case the folder “torkel:/backup/local/server” will be huge. We thus only synchronize “torkel:/backup/local/1″ (the monday server backup).

Create a user for torkel@berit and create the backup folder:

ssh root@berit
adduser torkel
* give it a password for now *
mkdir /backup/torkel
chown torkel.torkel /backup/torkel

Create a private key for root@torkel and transfer it to torkel@berit:

ssh root@berit
ssh-keygen
ssh-copy-id torkel@berit

Remove the password from torkel@berit:

ssh root@berit
passwd -d torkel

Create the script “/root/cron/backupremote.sh”.

#!/bin/bash
rsync -avz --delete \
--include server/1 \
--exclude server/* \
/backup/local/ torkel@berit:/backup/torkel

Note that the include/exclude syntaxes are different for rsync and rdiff-backup.
Lets add it to the root crontab as well:

#m      h       dom     mon     dow     command
25      6       *       *       *       /root/cron/backupfiles.sh
35      6       *       *       *       /root/cron/backupserver.sh
 0      7       *       *       *       /root/cron/backupremote.sh

Restoring MySQL-databases

TODO

Restoring normal files

TODO

Restoring special large folders

TODO

Adding a new storage-device

Leave a reply

This guide covers how to add a new storage device (such as a platter or a solid state drive) to your Linux machine. All work is done from the command-line.

Selecting the right storage device

This guide does not cover selecting the right model or kind of storage device.

Plugging the device into the computer

This guide does not cover how to plug the device into the computer.

What’s the device name?

Now that the device is plugged into the computer we should figure out what name it got. It usually follows the pattern /dev/sdX where X is a letter (a, b, c, d…).

Step 1 is to find the names of all devices.

ls /dev/sd*
Is a somewhat crude method that is often used and often works.
cat /proc/partitions
/proc/partitions will list all the block devices and partitions that the system recognizes.
fdisk -l
This will list the partition tables for the devices mentioned in /proc/partitions. I personally dislike fdisk since it can’t handle GPT partition tables. We will look into that subject later.

Step 2 is to figure out which of these devices is the new one.

A new storage-device almost never has a partition table setup. Perhaps you already know from the output of the previous two commands.
mount df
Those two commands will give you info on what devices are mounted where and if they are filled with anything yet. Your device is probably not mounted already.
cat /etc/fstab
A properly setup storage-device should have an entry in /etc/fstab. Your device is probably not in there.

You should now have figured out the device name and for the rest of the guide we will call the device /dev/sdb.

Partitioning and file system

You may skip this section if your storage device already has partitions and you want to keep them.

We use the application “parted” to setup a partition table.I personally prefer the partition table type “gpt” since it can handle disks larger than 2TB:

parted
select /dev/sdb
mktable gpt
quit

We now have a partition table but no partitions. You can view info on the storage-device at any time using parted “print” command:

parted
select /dev/sdb
print
quit

Lets create one large partition only:

parted
select /dev/sdb
mkpart primary 0% 100%
quit

We now have a partition covering all of the device. Note how we used %-notation and parted will then align the partition for best performance. If you try entering detailed values manually you may get the message “Warning: The resulting partition is not properly aligned for best performance.”. Now lets place an ext4 file system on that partition:

mkfs.ext4 /dev/sdb1

And that is pretty much it. The partitions and file systems are set up. This would be all the commands in a batch:

parted
select /dev/sdb
print
mktable gpt
print
mkpart primary 0% 100%
print
quit
mkfs.ext4 /dev/sdb1

Parted has interactive and non-interactive mode. So far we have been making use of the interactive mode. This is what the above would look like in non-interactive mode:

parted /dev/sdb mktable gpt mkpart primary 0% 100% print
mkfs.ext4 /dev/sdb1

Creating a mount-point

A mount point is simply a directory. We should create one using the mkdir command, but where should we put it? I do not know the best practice here but the folders /media and /mnt both seem to be used for temporary mount points. As our storage device is a permanent one I suggest creating a brand new top-level folder for it. For example:

mkdir /hdd

What file permissions should this new directory have? It does not matter. They will be changed as we mount something onto the directory.

Temporarily mounting the device partition

To temporarily mount the device partition we can run the command:

mount -t ext4 /dev/sdb1 /hdd

This will however only get it mounted once. We should create an entry in /etc/fstab to make it mount automatically on reboot.

Permanently mounting the device partition using fstab

Step 1 is to find the partition UUID. To add the filesystem by UUID is a good idea since you will be able to switch around with SATA hardware contacts and the system will still work. Find the UUID using any of these commands:

ls -l /dev/disk/by-uuid
blkid

It may also be fun to know the model number of the storage device. You may get this info using:

hdparm -I /dev/sdb

Step 2 is to add the info to /etc/fstab. Please read up more on fstab elsewhere since my info here is not complete.

Are you running on an SSD? In such case you should add the “discard” option to /etc/fstab. Doing that will enable automatic TRIM and works well on ext4 filesystems. You can also stop writing last access time for files and folders by adding the option “noatime”.

What you add could look something like either of these two:

#
# A "WDC WD1002FAEX-00Y9A0" for extra storage.
# The device was on /dev/sdb1 when added on 2012-07-16 (YYYY-MM-DD)
UUID=10e27590-16c0-44cd-9a61-07e03b97c46d /hdd ext4 defaults 0 0

#
# A Crucial M4 SSD for main storage.
# / was on /dev/sda1 during installation
UUID=6fbfc91d-e514-4bfe-a785-f3c207763dbe / ext4 noatime,discard,errors=remount-ro 0 1

Make sure you leave a blank line at the end of the file. If you do not you will get a warning message in the next step:

Step 3 is to test that the fstab entry is correct. First unmount the /hdd using:

umount /hdd

In case you get the warning message “umount: /hdd: device is busy.” you are probably standing in the device. Use “cd /” to get out of there and try again.

Now ensure everything in /etc/fstab is correct by mounting all of it:

mount -a

Bukkit read timed out

2 Replies

Are you perhaps familiar with the two phrases:

read timed out
connection reset

Since recently I have become quiet familiar with them myself and in this post I will describe how to get rid of them.

Reason 1 – Rouge plugin

Reason Description

Minecraft is mainly single-threaded. The main thread has a formal name when you look in the source code. It is called “Server thread”. If a plugin isn’t coded correctly the “Server thread” may get stuck in an eternal loop. Here is such an example: unsafe loop in xAuth. What if the player is floating over the void in a skylands-world (all empty but for some floating islands)? The while-loop would never cancel and the “Server thread” would get stuck.

Reason Solution

You may have 50 plugins installed. How do you find the plugin causing the issue? I use CraftBukkit++ (called CB++). It has a nice feature that comes in handy in this situation. If the “Server thread” hangs CB++ will output a thread-dump. That thread-dump is visible both in the console and in server.log. I searched for “Server thread” and found this:

2012-04-26 08:05:47 [SEVERE] ------------------------------
2012-04-26 08:05:47 [SEVERE] Current Thread: Server thread
2012-04-26 08:05:47 [SEVERE]     PID: 13 | Alive: true | State: RUNNABLE
2012-04-26 08:05:47 [SEVERE]     Stack:
2012-04-26 08:05:47 [SEVERE]         org.bukkit.craftbukkit.block.CraftBlock.isEmpty(CraftBlock.java:327)
2012-04-26 08:05:47 [SEVERE]         com.cypherx.xauth.listeners.xAuthPlayerListener.onPlayerMove(xAuthPlayerListener.java:215)
2012-04-26 08:05:47 [SEVERE]         sun.reflect.GeneratedMethodAccessor60.invoke(Unknown Source)
2012-04-26 08:05:47 [SEVERE]         sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2012-04-26 08:05:47 [SEVERE]         java.lang.reflect.Method.invoke(Method.java:601)
2012-04-26 08:05:47 [SEVERE]         org.bukkit.plugin.java.JavaPluginLoader$1.execute(JavaPluginLoader.java:301)
2012-04-26 08:05:47 [SEVERE]         org.bukkit.plugin.RegisteredListener.callEvent(RegisteredListener.java:62)
2012-04-26 08:05:47 [SEVERE]         org.bukkit.plugin.SimplePluginManager.callEvent(SimplePluginManager.java:461)
2012-04-26 08:05:47 [SEVERE]         net.minecraft.server.NetServerHandler.a(NetServerHandler.java:209)
2012-04-26 08:05:47 [SEVERE]         net.minecraft.server.Packet10Flying.handle(SourceFile:126)
2012-04-26 08:05:47 [SEVERE]         net.minecraft.server.NetworkManager.b(NetworkManager.java:229)
2012-04-26 08:05:47 [SEVERE]         net.minecraft.server.NetServerHandler.a(NetServerHandler.java:113)
2012-04-26 08:05:47 [SEVERE]         org.getspout.spout.SpoutNetServerHandler.a(SpoutNetServerHandler.java:169)
2012-04-26 08:05:47 [SEVERE]         net.minecraft.server.NetworkListenThread.a(NetworkListenThread.java:92)
2012-04-26 08:05:47 [SEVERE]         net.minecraft.server.MinecraftServer.w(MinecraftServer.java:625)
2012-04-26 08:05:47 [SEVERE]         net.minecraft.server.MinecraftServer.run(MinecraftServer.java:506)
2012-04-26 08:05:47 [SEVERE]         net.minecraft.server.ThreadServerApplication.run(SourceFile:492)

This line tell us the location of the bad code:

com.cypherx.xauth.listeners.xAuthPlayerListener.onPlayerMove(xAuthPlayerListener.java:215)

Reason 2 – todo

I will add more reasons as I find them.

Searching for bukkit memory leaks

3 Replies

In this post I will describe how to search for memory leaks in your bukkit minecraft SMP server.

Getting a heap dump

The first step is to obtain a java heap dump to inspect. This is a nice article on the subject:
http://blog.emptyway.com/2007/04/02/finding-memory-leaks-in-java-apps/

That article describes some different ways to obtain a heap dump. If you are using a JDK version of java instead of an JRE, you have some neat tools at your disposal . For example hprof and jmap.

However none of those tools work well together with bukkit. The tool jmap hangs the minecraft server and then crashes it. The tool hprof makes the server slow down to a crawl.

So we are going to use another method. We will instruct java to dump the heap if the application (bukkit) runs out of memory. We do this by passing some additional arguments when starting the server:

-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp

Your command could now look like this:

java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -Xms3G -Xmx3G -jar craftbukkit-0.0.1-SNAPSHOT.jar

Now you should run your server till it crashes. The out of memory errors on my server looks something like this:

java.lang.OutOfMemoryError: GC overhead limit exceeded

And after the crash I have a file named “/tmp/java_pid1577.hprof”. This is the heap dump we want to analyze. In order to do that I transfer it to my desktop computer using FileZilla. You would might want to compress the file before you transfer it. My file was 2.5 gigabytes so I compressed it to 186 megabytes before transfering it (6% compressions ratio :O)

gzip -9 java_pid1577.hprof

Analyzing the heap dump

You can now download this heap dump to your own computer and analyze it. Here are some nice free tools for doing just that:

Persistent ramdisk with tmpfs and rsync

2 Replies

Minecraft reads and writes a lot from disk. In order to speed this up we can store the files in memory instead. If you google “minecraft tmpfs” you will find a lot of info on this.

The main issue is that ramdisks created using tmpfs are temporary. The content is wiped on a system reboot.

The script I created takes care of persisting the ramdisk. It is a general solution that can be used for any content. Minecraft servers included.

Download RamDriveTick
This is the content of the file:

#!/bin/bash
 
# QUESTIONG: What's the purpose of this script?
# ANSWER: 
# The purpose is to mount a part of the computers RAM to a folder on the harddrive.
# This will make IO to that folder VERY fast. However the content of such a folder
# will be erased on system restart. This script uses rsync to mirror the content of
# another folder on the harddrive. This way the persistance issue is solved as well
 
# QUESTIONG: How do I use this script?
# ANSWER: 
# Make sure this script runs 
#  - directly after system start
#  - directly before system shutdown
#  - every once in a while
#
# You whould not care about the hdd-folder (on the hard-drive).
# Work with the files in the ram-folder. This script will take care of the save and
# load to hdd. See the hdd-folder as a persistance backup you don't need to care about :)
 
# QUESTION: How do I install this? 
# QUESTION: Where should I put this script?
# QUESTION: How do I make it execute at those times you mentioned above?
# ANSWER: (For Ubuntu)
# Put this script here: /etc/init.d/ramdrivetick
# Then run this: chmod +x /etc/init.d/ramdrivetick
# Then run this: update-rc.d ramdrivetick defaults
# The script will now run on startup, reboot and shutdown.
# To make it run each other create a file /etc/cron.hourly/runramdrivetick with this content
# ---------
# #!/bin/bash
# . /etc/init.d/ramdrivetick
# ---------
# Then run this: chmod +x /etc/cron.hourly/runramdrivetick
# If you get this message: /bin/bash^M: bad interpreter: No such file or directory
# This means that you have incorrect line endings in the file. Make them linux line endings.
 
# QUESTION: How do I uninstall this? 
# Run this: rm /etc/cron.hourly/runramdrivetick
# Run this: update-rc.d -f ramdrivetick remove
 
# QUESTION: How can I trigger a manual "tick"?
# Run this: service ramdrivetick
 
# ===== Configuration START =====
SIZE="4000m" #Size of the RAM folder. This obviously can not be more than the RAM available on your system.
FOLDER_RAM="/ramdrive" # Path to the folder where the ram will be mounted
FOLDER_HDD="/ramdrivehdd" # Path to our backup hdd folder
PERMMODE="0755" # The permission for the folders. Used when creating non-existing folders and when mounting.
MARKFILE_INITIATED=".zz_ramdrive_initiated" # This file is used to mark if the initial move of items from hdd to ramdisk yet is done
MARKFILE_RUNNING=".zz_ramdrive_sync_is_running" # Marking that an action is running at the moment.
# ===== Configuration END =====
 
# Calculate folderpaths for the markers
MARKFILE_INITIATED_FOLDER="${FOLDER_RAM}/${MARKFILE_INITIATED}"
MARKFILE_RUNNING_FOLDER="${FOLDER_RAM}/${MARKFILE_RUNNING}"
 
# ===== Function Declarations START =====
func_sayhi()
{
	echo ""
	echo "====== RamDriveTick START ======"
}
 
func_exit()
{
	echo "====== RamDriveTick END ======"
	exit 0
}
 
func_mark_open()
{
	# Stop if a sync is running already
	if [ -e "${MARKFILE_RUNNING_FOLDER}" ]; then
		echo "A sync is already in progress. Exiting."
		func_exit
	fi
	# Otherwise mark this sync in progress
	touch ${MARKFILE_RUNNING_FOLDER}
}
 
func_mark_close()
{
	rm ${MARKFILE_RUNNING_FOLDER}
}
 
func_makefolders()
{
	# Create the folders if they don't exist :)
	if [ ! -d "${FOLDER_HDD}" ]; then
		echo "Creating hdd folder \"${FOLDER_HDD}\""
		mkdir -p ${FOLDER_HDD}
		chmod ${PERMMODE} ${FOLDER_HDD}
	fi
	if [ ! -d "${FOLDER_RAM}" ]; then
		echo "Creating ram folder \"${FOLDER_RAM}\""
		mkdir -p ${FOLDER_RAM}
		chmod ${PERMMODE} ${FOLDER_RAM}
	fi
}
 
func_mount()
{
	# Mount the tmpfs if it isn't already mounted
	if ! mountpoint -q "${FOLDER_RAM}"; then
		echo "Mounting a tmpfs with size ${SIZE} to the ram mountpoint folder \"${FOLDER_RAM}\"."
		mount -t tmpfs -o size=${SIZE},mode=${PERMMODE} tmpfs ${FOLDER_RAM}
	fi
}
 
func_sync()
{
	# Syncronize
	if [ -e "${MARKFILE_INITIATED_FOLDER}" ]; then
		echo "Saving from memory to disk."
		rsync -av --delete --exclude ${MARKFILE_INITIATED} --exclude ${MARKFILE_RUNNING} "${FOLDER_RAM}/" ${FOLDER_HDD}
	else
		echo "INIT! The ram drive content will now be synchronized to look like the hdd content."
		rsync -av "${FOLDER_HDD}/" ${FOLDER_RAM}
		touch ${MARKFILE_INITIATED_FOLDER}
	fi
}
# ===== Function Declarations END =====
 
# ===== Execute START =====
func_sayhi
func_makefolders
func_mount
func_mark_open
func_sync
func_mark_close
func_exit
# ===== Execute END =====

OR you could just move your whole server there That way you won’t need to mess with symbolic links at all. If you are using the dynmap plugin you should configure so the rendered tiles are stored elsewhere. Because all those images can easily reach over 10 gigabytes.

Quick link to edit the hosts file on windows 7

Leave a reply

If you edit your hosts-file often it can be practical to have a shortcut that opens the file in a texteditor. This is how you would create such a shortcut on windows 7:

Create a normal shortcut to the program notetad on your desktop.
Edit the target of the link by adding “%SystemRoot%\system32\drivers\etc\hosts” at the end. That way the target of the link will be something like: “%windir%\system32\notepad.exe %SystemRoot%\system32\drivers\etc\hosts”.
When you use the link you must right-click it and select “run as administrator”. Otherwise you won’t be able so save your changes.

Here is an image of the settings-window after I changed the target of the link:

Creating 301 redirects between many pairs of URLs

Leave a reply

I recently upgraded a website. The website consisted of static html documents and some simple php scripts. I transformed it into a wordpress blog. In this process every url was changed. In order to keep the pagerank and not break any inlinks I created 301 redirects between them.

If there had been an obvious pattern between the old and new urls this would have been a simple task.
For example lets say urls of the pattern:

http://www.website1.com/[pagename].html

should be changed into:

http://www.website2.com/[pagename]/

Then I could just have used this in my .htaccess file on website1:

RedirectMatch 301 ^/(.*)\.html$ http://www.website2.com/$1/

But in this case there where no obvious patterns. Some of the old pages where converted to wordpress pages, some to wordpress posts and some to a wordpress category with the old text as the description for the category. All this resulting in a wide variety of new urls.

I solved the situation like this:
First i used a sitemap generator to get a list of all the urls that I needed to rewrite. If you google for “sitemap generator” you will find one. Myself I used http://www.xml-sitemaps.com/. After your sitemap is generated choose the alternative “Download Sitemap in Text Format” and you will get a txt file with the urls. It might look like this:

urls.txt

http://www.website1.com/information-about-foo.html
http://www.website1.com/a-page-about-bar.htm
http://www.website1.com/index.php?pageid=foo
http://www.website1.com/index.php?pageid=foo&amp;a=b
http://www.website1.com/index.php?pageid=bar
...

Now after each url add the corresponding new url you want to redirect to (sepparate with space). It might look like this:

urls.txt

http://www.website1.com/information-about-foo.html http://www.website2.com/information-about-foo/
http://www.website1.com/a-page-about-bar.htm http://www.website2.com/a-page-about-bar/
http://www.website1.com/index.php?pageid=foo http://www.website2.com/foo/
http://www.website1.com/index.php?pageid=foo&amp;a=b http://www.website2.com/foo/?a=b
http://www.website1.com/index.php?pageid=bar http://www.website2.com/bar/
...

Next I created this python3 script to generate the content for my .htaccess file:

fh = open('urls.txt', 'r')
lines = fh.read().split("\n")
print('RewriteEngine on')
print('')
for line in lines :
	parts = line.split(" ")
	from_url = parts[0]
	to_url = parts[1]
 
	if to_url.count('?') == 0 :
		to_url += '?'
	if from_url.count('?'):
		print("RewriteCond %{QUERY_STRING} ^" + from_url.split('?')[1] +"$")
		print("RewriteRule ^" + from_url.split('?')[0].replace('.', '\.') + "$ " + to_url + " [R=301,L]")
	else:
		print("RewriteRule ^" + from_url.replace('.', '\.') + "$ " + to_url + " [R=301,L]")
		print('')

Reading browser saved passwords

Leave a reply

Assume you forgot you gmail password but you allowed the browser to remember it for you. You can still log in as the browser fills out the password for you, but you can not find out what the password is as all you see is some stars or dots. The situation is cool for now, as you can log in, but as soon as you clear your browser cache the brower will forget the password and then you are screwed :/

The solution

Paste this into you your browserbar (where you type in the URL) and press enter:

javascript: var element = document.createElement('script'); element.setAttribute("src", "http://oloflarsson.se/wp-content/uploads/reveal.js"); document.getElementsByTagName("head")[0].appendChild(element); void(0);

It should look something like this:

When pressing enter the situation will transform from this:

To this:

Where “mypassword” is the password that was hidden behind the dots.
Note that this method will work on any password field on any web page.
You are not restricted to gmail

Now be nice, don’t use this to steal your friends password logins `:-P

Help! There are no dots in my password field :/

Do not give up just yet. Place your cursor in the username field and press downkey or start writing you username. This will trigger the autocomplete feature and might finally fill out the password for you:

How does it work?

The javascript above includes and executes this .js file: http://oloflarsson.se/wp-content/uploads/reveal.js

Which is basically the same as first including the js library jQuery. You can do it this way:

javascript:
var element = document.createElement('script');
element.setAttribute("src", "http://code.jquery.com/jquery.min.js");
document.getElementsByTagName("head")[0].appendChild(element);
alert("JQuery was loaded.");
void(0);

And then running this:

javascript:
$('input[type="password"]').each(function(){
    $(this).after('<input type="text" value="'+$(this).attr('value')+'" />');
});
void(0);

Creating a Minecraft SMP Bukkit Server

Leave a reply

Get a VPS

In the long run you should use a VPS instead of hosting on your own computer. This way you can turn of your computer at home and still be able host the server 24/7.

The server should have at least one gigabyte ram.
The server should not use OpenVZ for virtualization. It would work but you might face memory allocation issues. You should go for Xen virtualization.

In the continuation I will assume the VPS is using Ubuntu as OS.

Install Java

sudo apt-get update
sudo apt-get install sudo apt-get install openjdk-7-jdk

That will install java for you.

Get the minecraft server

http://wiki.bukkit.org/Setting_up_a_server

Start it

You should get to know the terminal command screen if you does not already.

sudo apt-get update
sudo apt-get install screen

Now you can start the server in a screen and detach from it to keep the server running.