Comments on: SQL Management Studio… How NOT to save in Unicode format

By: Kevin

Kevin — Sat, 28 May 2011 01:29:43 +0000

This is not a rare problem at all. Only people who never try to access these files outside the confined studio env won’t be hitting the froblem. Anyone who uses other tools (including many Microsoft tools) to search (e.g. grep) or diff such source files would have this problem.

Microsoft in their typical non-interoperable way picks a default that doesn’t make much sense, not just for other environments, but doesn’t make sense, period.

For starters, calling UTF-16 ‘Unicode’ is wrong. The Microsoft encoding default is UTF-16 which is one specific form of Unicode. UTF-8 is also Unicode, see: http://en.wikipedia.org/wiki/UTF-8. Ironically, UTF-8 is much more compatible with ASCII so it would be a much more sensible default to pick.

The problem with the default Microsoft encoding is that for most cases (English text) every other byte in the UTF-16 encoded file is a null (ASCII 0) byte. This makes these *.sql files both 2 times larger for no reason and non trivial to search, diff, edit with other standard tools.

Yet another Microsoft bad design decision.

By: Jonas Porter

Jonas Porter — Tue, 27 Apr 2010 18:47:45 +0000

Man, that’s a tricky ‘Save as’ button. Thanks for the information.

By: Kelly

Kelly — Wed, 19 Aug 2009 10:35:12 +0000

I’ve never written a SQL add-in before and I think it would be really silly to write an add-in to replace the editor, but since MS have not given us the ability to hook the save event in the editor, nor the ability to set the default encoding for the editor, this may be the only way to get the type of behaviour desired.

By: TW Burger

TW Burger — Tue, 28 Jul 2009 23:52:16 +0000

Thanks Chris,

If Microsoft can do anything in a more difficult, counterproductive, undocumented, hard to find manner – they will.

I can sort of understand saving in Unicode as a default, but why won’t Excel read the Unicode correctly and separate the values a it would with an ASCII file? Sheesh!

By: Yakov

Yakov — Tue, 05 May 2009 22:35:55 +0000

Besides, Unicode makes it impossible to search for files with find/grep!

By: PH

PH — Mon, 27 Apr 2009 18:29:40 +0000

Just keeping the conversation alive. We have the EXACT same problem, and the feedback you posted is still classified as “Active”. I have the sneaking suspicion that M$ won’t do anything to “correct” this because it would result in making their product work with competitors’ source repository solutions. I hate Harvest, but I have to use it.

By: JH

JH — Fri, 21 Nov 2008 11:04:10 +0000

Hi,

I’ve had exactly same problem and your solution works… a bit.
The trouble is our database uses Czech collation & czech encoding, but when I chose ANSI in Save As dialog, I get questionmarks all over the data, replacing anything that’s not pure 7bit. Unicode is not really an option because of applications that use exported files and Chinese doesn’t work either.

Any ideas, how to get any othe codepage to the list?

By: DG

DG — Fri, 11 Apr 2008 19:32:59 +0000

Thanks Chris!

I added your step by step instructions to the post as well as your link to the SQL feedback post. Hopefully it will help someone who *gasp* is developing SQL scripts in a team environment */gasp* get their code into source control and functional.

With Microsoft’s supposed focus on making their tools developer-centric over the past 5 years (with the tools we have now as the result of that initiative, anyone remember Ballmer’s “Developers, developers, developers, developers” scene?) it surprises me that they aren’t taking a more pro-active approach to making sure team development is possible and intuitive.

By: Chris May

Chris May — Fri, 04 Apr 2008 01:18:57 +0000

I have found some information about this.
Though it is possible at the time a script is saved to change the encoding to ascii it is tedious. Here is the process.
1) Choose File\Save … As
2) Choose the name and folder to save the file then look really closely at the right edge of the “Save” button for a tiny arrow.
3) Click that tiny Arrow and choose “Save with Encoding”
4) From the Drop list select the encoding you want (the default encoding is “Unicode – Codepage 1200″, which means “UTF-16″). I have been using “US-ASCII – Codepage 20127″.
5) Hit OK and Save. Your files should now work just fine with Perforce, CVS, etc.

There is not currently (or as far as I know planned) a way to change the default encoding for these files, but I have submitted a feedback to the SQL team and would request anyone affected by this issue to add their voice so M$ knows how important this is. Here is the URL of my feedback:
https://connect.microsoft.com/SQLServer/feedback/ViewFeedback.aspx?FeedbackID=336750

By: MDS

MDS — Thu, 14 Feb 2008 16:40:48 +0000

This is also a problem with Microsoft’s legacy systems. We’re still using -ugh- visual source safe (VSS). VSS saves unicode files as binary. One only realizes there is a problem after running a VSS comparison of the two files. VSS comparisons of binary files only give one the helpful information that “the files differ.”

I’m still looking for a way to set the default but I’m thankful that at least I can now save the file to ASCII without cutting and pasting into notepad. The tip is much appreciated.