SQL Management Studio… How NOT to save in Unicode format
A project I work on requires that we developers edit stored procedures and store the resulting script in a text file that we put in our source control tool. In our case, we use Microsoft SQL Server Management Studio to do the aforementioned editing and we use CA’s Harvest Change Manager as the corporate mandated code repository and source control tool.
Now if you were using another tool, Notepad or Query Analalyzer, for instance, you probably wouldn’t have the problem I’m going to outline. As a matter of fact, the problem I’m going to outline is probably pretty rare. But it exists and I found a solution, so I’m going to write about it.
As it turns out, whenever you “Save As” in Microsoft SQL Server Management Studio the default encoding for the file is Unicode – Codepage 1200. Yes, this is a new approach to saving your beloved stored procedures and no, it wasn’t done that way in the past. Further, I can’t find any notification that the “standard default” was going to be changing.
Anyway, Harvest doesn’t like Unicode files. So when you go to add the file to your Harvest project it won’t let you. Oh, the humanity! ANSI for everyone!
Now I may be in the minority here, but I originally failed to notice that on the “Save File As” dialog there is a small arrow on the right side of the [Save] button. Once I noticed it, I clicked it… Then I clicked “Save with encoding…” Now I’m happy again! Kind of…
The good thing is that this will let you save your file in whatever encoding you want. The caveat is that it will let you save your file in whatever encoding you want.
So how is the default set? I’ll leave that for another article… Mostly because I don’t know yet.
Update (04.11.2008): Thanks to Chris May for the following step by step instructions on how to overcome this issue (edited for formatting, the original version is comment #8):
I have found some information about this.
Though it is possible at the time a script is saved to change the encoding to ascii it is tedious. Here is the process.
- Choose File\Save
- Choose the name and folder to save the file then look really closely at the right edge of the “Save” button for a tiny arrow
- Click that tiny Arrow and choose “Save with Encoding”
- From the Drop list select the encoding you want (the default encoding is “Unicode – Codepage 1200″, which means “UTF-16″). I have been using “US-ASCII – Codepage 20127″
- Hit OK and Save. Your files should now work just fine with Perforce, CVS, etc.
Additionally, Chris has voiced his feedback to the SQL team, I encourage everyone to go have a look and hopefully we can get a solution sooner rather than later… although the outlook is bleak.
about 4 months ago
Man, that’s a tricky ‘Save as’ button. Thanks for the information.
about 1 year ago
I’ve never written a SQL add-in before and I think it would be really silly to write an add-in to replace the editor, but since MS have not given us the ability to hook the save event in the editor, nor the ability to set the default encoding for the editor, this may be the only way to get the type of behaviour desired.
about 1 year ago
Thanks Chris,
If Microsoft can do anything in a more difficult, counterproductive, undocumented, hard to find manner – they will.
I can sort of understand saving in Unicode as a default, but why won’t Excel read the Unicode correctly and separate the values a it would with an ASCII file? Sheesh!
about 1 year ago
Besides, Unicode makes it impossible to search for files with find/grep!
about 1 year ago
Just keeping the conversation alive. We have the EXACT same problem, and the feedback you posted is still classified as “Active”. I have the sneaking suspicion that M$ won’t do anything to “correct” this because it would result in making their product work with competitors’ source repository solutions. I hate Harvest, but I have to use it.
about 1 year ago
Hi,
I’ve had exactly same problem and your solution works… a bit.
The trouble is our database uses Czech collation & czech encoding, but when I chose ANSI in Save As dialog, I get questionmarks all over the data, replacing anything that’s not pure 7bit. Unicode is not really an option because of applications that use exported files and Chinese doesn’t work either.
Any ideas, how to get any othe codepage to the list?
about 2 years ago
Thanks Chris!
I added your step by step instructions to the post as well as your link to the SQL feedback post. Hopefully it will help someone who *gasp* is developing SQL scripts in a team environment */gasp* get their code into source control and functional.
With Microsoft’s supposed focus on making their tools developer-centric over the past 5 years (with the tools we have now as the result of that initiative, anyone remember Ballmer’s “Developers, developers, developers, developers” scene?) it surprises me that they aren’t taking a more pro-active approach to making sure team development is possible and intuitive.
about 2 years ago
I have found some information about this.
Though it is possible at the time a script is saved to change the encoding to ascii it is tedious. Here is the process.
1) Choose File\Save … As
2) Choose the name and folder to save the file then look really closely at the right edge of the “Save” button for a tiny arrow.
3) Click that tiny Arrow and choose “Save with Encoding”
4) From the Drop list select the encoding you want (the default encoding is “Unicode – Codepage 1200″, which means “UTF-16″). I have been using “US-ASCII – Codepage 20127″.
5) Hit OK and Save. Your files should now work just fine with Perforce, CVS, etc.
There is not currently (or as far as I know planned) a way to change the default encoding for these files, but I have submitted a feedback to the SQL team and would request anyone affected by this issue to add their voice so M$ knows how important this is. Here is the URL of my feedback:
https://connect.microsoft.com/SQLServer/feedback/ViewFeedback.aspx?FeedbackID=336750
about 2 years ago
This is also a problem with Microsoft’s legacy systems. We’re still using -ugh- visual source safe (VSS). VSS saves unicode files as binary. One only realizes there is a problem after running a VSS comparison of the two files. VSS comparisons of binary files only give one the helpful information that “the files differ.”
I’m still looking for a way to set the default but I’m thankful that at least I can now save the file to ASCII without cutting and pasting into notepad. The tip is much appreciated.
about 2 years ago
It’s a big problem for my team. Perforce won’t handle unicode files either. So far my searching for a solution has only led me here.
about 2 years ago
@SY
I would recommend one of the ASCII formats to be sure the file would work. There may be another format which would work better for your situation, but for source code written in English generally ASCII or ANSI formats seem to work fine.
In response to your second comment, I still haven’t found a “fix” for this which would allow you to change the default save format. I’m hoping someone will eventually stop by with that information, but they haven’t yet.
about 2 years ago
Also, is there a known solution for this yet? I can’t seem to find it on the web.
about 2 years ago
I too am experiencing a same problem with ClearCase as the user above. Do you know what “encoding” I have to save as to circumvent this problem?
about 2 years ago
I’m… in one way… glad to hear that I’m not the only person who has come across this. I’ve had other people who have contacted me privately with the same issue, but you’re the first to comment here about it.
When I originally researched this problem Microsoft made no claims as to why the change was made (although I have since noticed a similar change in other software they provide) and there was no information given on how to change the default.
If you come across one, please post it here in another comment and I’ll update the original post.
about 2 years ago
FYI – We have the exact same problem with Rational ClearCase…so it may not be a really rare problem.
In our ClearCase implementation, we can add the files, but they do not always merge correctly, as they are thought to be binary files.
I’m still looking for a way to set the default-