exchange 2010

Exchange store failes due to AD topography changes

So today I was working at a site that has a single Exchange 2010 server that forfills all the Exchange roles (I know….) which happened to fall on its ass.

First thing I did was check the Exchnage services which were in a state of “starting” which is never good and then I went to the registry and found:

MSExchange ADAccess, EventID 2141
Process STORE.EXE (PID=2996). Topology discovery failed, error 0x8007077f

MSExchange ADAccess, EventID 2142

Process MSEXCHANGEADTOPOLOGYSERVICE.EXE (PID=1760). Topology discovery failed, error 0x8007077f

Here’s a few screen shots:

exchange error1exchange error2exchange error3exchange error4

As well as a few more related to AD.

After some investigation I found out that a new DC in a new site had been created for some DFS replication amongst other things.

As the system could start the Microsoft Exchange Active Directory Topology service (until it failed and is restarted by dependent services), Exchange’s other services were also triggered, leading to almost indefinitely restarting services as configured in their corresponding service recovery actions sections.

So next up is to look at Active Directory Sites and Services:

exchange error5

And as you can see from the screen shot the subnets are missing, which is going to cause issues as the new DC is on a different subnet.

When Exchange can’t determine in which site a computer belongs, the function DSGetSiteName, used to retrieve the current site, returns an error 1919 0x77f (ERROR_NO_SITENAME) which in turn kills off Exchange.  You can test this by running nltest /dsgetsite in a command prompt or by having a look at  HKLM\System\CurrentControlSet\Services\Netlogon\Parameters\DynamicSiteName.

To solve the issue you can do any of the following:

  1. Making the site association static using a registry key, which isn’t a best practice. If you must, set registry key HKLM\SYSTEM\CurrentControlSet\Services\Netlogon\Parameters\SiteName(REG_SZ) to the desired site name;exchange fix1
  2. Adding proper subnet definitions;exchange fix2
  3. Remove the new site.

 

Finally give Exchange a  bounce and BOOM.

 

Note that the NetLogon service determines site association membership at startup and every 15 minutes. The Microsoft Exchange Discovery Topology service maintains this information by caching the information in the msExchServerSite attribute of the Exchange server object, in order to reduce load on active directory and DNS. Therefore, you might need to wait or restart Microsoft Exchange Discovery Topology if you want to renew site association membership.

 

 

Outlook Anywhere with Additonal UPN suffix

Posted by robd on February 03, 2015
Active Directory Sites and Services, exchange 2010 / 1 Comment

So today we had a requirement to start using Outlook Anywhere (Outlook over RPC) on our Exchange 2010 server, the setup is dead easy:

Logon to your CAS server,

Install the server feature “RPC over HTTP Proxy”,

Open the Exchange Management consol and go to Server Configuration and Client Access, Click Enable Outlook Anywhere,

Use your domain (you dont need HTTP/S), choose your authentication and if you need to offload your certificate to another server:

Website1

Thats the Exchange bit done.

Next change your DNS:

Locally and externally you need two things; an A record and a SRV record both need to point at Autodicsover.DOMAIN.ORG, here’s two examples from 123 reg:

Website2 Website3

Finally make sure 443 is allow through your Firewall.

Last but not least test with:

https://testconnectivity.microsoft.com

Ok so now onto the UPN bit, in my case my email differed from my domain i.e. my domain is bohemian.local and my email domain is bohemiangrove.co.uk.

What this means is when logging on via Outlook I get prompted to type domain and password:

hmsg1hmsg2 hmsg4

If you’d rather not do this then you need to add your email domain as a UPN suffix.

To do this, on a DC open: Active Directory Domains and Trusts

rom the displayed context menu, click Properties.

1

On the properties box that appears, in the Alternative UPN suffixes field, specify the desired alternate UPN suffix for the domain and click Add.

2

Apply and ok. Finally open Active Directory Users and Computer

Find a user and under User logon name section, choose the alternate UPN suffix from the drop-down list that was created earlier in Active Directory Domains and Trusts snapin.

3

 

Then test Outlook Anywhere using the users email as the username.

 

Tags: , ,

Service Pack 3 for Exchange 2010

Service Pack 3 for Exchange 2010

So I only recently got around to installing SP3 for Exchange 2010 (I’ve been on holiday for a few weeks and what not) and thought I’d share my experience around the experience!

Now you may or may not know this service pack updates the Active Directory Scheme which basically means when you install this update you must be a Schema or Enterprise Admin to run the installation!!

Now if like me your Exchange Server sits on a completely separate domain to your Root Active Directory servers you’ll need to do a little planning.

For example I have a domain, let’s call it BohemianGrove.co.uk and a sub domain called Corp.BohemianGrove.co.uk.  BohemianGrove.co.uk is where my root Active Directory Domain Controllers are installed which means that’s where the Schema settings are stored for Exchange, these settings then filter down to sub domains which in my case is where my Exchange servers are installed i.e Corp.BohemianGrove.co.uk.

All this means in regards to this update is I’ll need to run Setup.com /prepareAD from BohemianGrove.co.uk as an Enterprise Admin before I run the update from the actual Exchange servers.

Exchange_Adprepare

So from the above you can see that all went very smoothly luckily for me (I won’t write about how I tried to run the SP2 update two or three times which produced soooo many errors)!  From here I waited 20 mins for the Schema update to replicate down, although you can force a update via Active Directory Sites and Services.

 

Next came the update for the servers but please note, install the update on the roles in this order:

Client Access servers,

Hub Transport (My HT and CAS roles are on the same server),

Edge Transport servers,

Mailbox servers,

Unified Messaging servers (I don’t have any, does anyone?),

 

Before I start talking about the install I should mention that you should only install the update on nodes or DAG servers that arnt active i.e. you need to fail an active server to a passive server and run the update, then fail back.  Here’s a very quick how to do this:

 

CAS/Hub Transport –

–  Stop the active node in the cluster via Network Load Balancing Manager by right clicking a node and click Clicking Control Host then Stop (or DrainStop if your worried users are connected still),

– In Network Load Balancing Manager right click the same node and choose properties and set the Default state to STOPPED, this will stop the node auto joining the cluser when once it reboots,

– Finally run the update, reboot the server and re-start the Default state.

– Do the above to the next node.

 

Mailbox Server  with a DAG –

–  Find the active database plus where the PAM role is and make a note:

–          Check the cluster group:

–          The above is just an initial check to ensure that the environment is in a situation where it is ready for manual failover.  The next set of instructions will fail the DB over to the passive server, prevent DB automounts, and reconfigure the cluster.

–          Run DAG Maintenance script (run from C:\Program Files\Microsoft\Exchange Server\V14\Scripts):

(please note overrideMinimumTwoCopies, this is because I only have two servers in my DAG).

–          This fails the active DB over to MB02. A quick check of the Management Console will show this is correct.

–          Rerunning the PAM holder and Cluster owner commands will show the second server as the master,

–          You can now conduct your tasks on this server with no Exchange downtime,

–          Once your tasks are complete, it’s time to fail the services back,

–          From Exchange Powershell, run; (From C:\Program Files\Microsoft\Exchange Server\V14\Scripts)

–          There is no output from this script as it simply allows us to now make changes. Fail databases back

–          Switch Cluster Master cluster group

–          Finally run both the PAM and Cluster query commands to ensure both roles are back with the active server,

 

So run the install:

Extract_Binary

 

Upgrade

Exchang Start

 

The install will check if you’ve updated the schema and if your servers, click Upgrade and the install will start, be warned it can take a while to install, mine took about 30 mins which seemed to be mostly the language pack:

Readiness

Install

Fortunately for me the install went smoothly on all my servers but just be careful to only run the update on servers that are not active.  Also note that once you update a server with a database on you cant fail that database to a none updated server.

Finally make sure you run Exchange 2010 SP3 Rollup 3 as this will fix a heap of issues with Exchange 2010 SP3.

Tags: , , , , ,

MAPI Connections to Exchange 2010

Posted by robd on August 19, 2013
exchange 2010, Office 2010, powershell / 5 Comments

So had a bunch of errors on my Exchange 2010 server today, namely:

Event ID 9646 MSExchangeIS

The error doesn’t give much away but basically what its saying is:

A user with GUID “ccd83bc42-1bff-459e-9db2-558cc68478b9” has more than 500 open folders in a single session, because exchange only allows 500 open folder connections (for a cached mailed) at a time its going to screw with the user and in our case not automatically update the users folders in Outlook 2010.  Please note this is different from open MAPI connections to the mailbox.

Note this has been edited with help from Mark Daley who stated:

the objtfolder is an ‘Open Item Limit’ ie the limit of the number of folders that are being opened in a single session, not the number of connections being made to the mailbox which is a session limit See (http://technet.microsoft.com/en-us/library/ff477612(v=exchg.141).aspx). This limit is the total for the session so if a user has 250 folders in their own mailbox and say 300 in a shared mailbox then the limit will be breached. The other thing that should be explained in that this is only likely to occur when the mailboxes are cached (shared and non-shared) as it is the consequence of caching that is causing the limit to be breached because each folder is opened / counted during the mailbox sync process. Therefore turning off caching mode for either the primary mailbox or just the shared mailbox is another fix for this issue.

 

You may be wondering how a user could have more than 500 connections, well in our case the user had two mailboxes attached to their own mailbox and all the folders (inbox, outbox etc) together added up to more than 500 folders.

You can scan a mailbox with Power Shell to determine the number of folders:

So lets say you see the errors on your Exchange 2010 server but don’t know who’s having the issue, you find Users MAPI GUID’s by running the following Power Shell command:

Once you know who the user is what can you do, well there’s four options in my mind:

1) Tell the user to re-organise the folder structure,

2) Remove some of the extra Mailboxes from a users mail profile if possible (you could add a extra profiles to Outlook, making Outlook prompt on start up),

3) Do not cache the mailbox in Outlook or simply do not cache the additional mailbox in Outlook.

4) Change Exchange 2010 to allow more than 500 folder to be open at a single time for cached mailboxes!

 

To proceed with 4) you’d need to do the below but be warned if large quantities of users connected with plus of 500 folders your server performance will suffer drastically:

Log onto all your mailbox servers and create the following registry key:

  1. Click Start, click Run, type regedit, and then click OK.
  2. Expand the following registry subkey:
    \\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MSExchangeIS\ParametersSystem
  3. Right-click ParametersSystem, point to New, and then click Key.
  4. Type MaxObjsPerMapiSession, and then press ENTER to name the new sub key.
  5. Right-click MaxObjsPerMapiSession, click New, and then click DWORD Value.
  6. Enter the object type as specified in the event description, and then press ENTER.
  7. Right-click the object type that you entered, and then click Modify.
  8. In the Value data box, type the number of objects to which you want to limit this entry, and then click OK.

 

Tags: , , , , , , , , , ,

Exchange 2010 Outlook Web App (OWA) – Logo Change

Posted by robd on May 03, 2013
exchange, exchange 2010, Outlook Web App, OWA / No Comments

So a customer recently had issues where their users were typing the wrong URL and then complaining that they couldn’t log in.

To try and help (and can you really help users who don’t know their own URL?, I mean come on you work there a minimum of 37 hours a week) this issue I suggested adding a simple logo to the OWA home page, i.e. go from:

3

To

4

So I logged onto each of the four Exchange 2010 Client Access Servers navigated to:

And then chose the folder that matched the version of Exchange in our case: 14.342.3 (to check open Exchange 2010 console and click help and About Exchange Server 2010):

5

Next I found the logo file I needed to modify, specifically:    lgntopl.gif

Edited the gif to look like the above (well not exactly like that) and replaced the on each Client access server.

Navigated to the web and BAM…… didn’t work!

So next I checked the exchange console > Server Configuration > Client access

6

And noticed the version was actually on Build 247.5, so decided to swap the gif in the matching location:

7

And BAM, still didn’t work!!

Well at this point I was annoyed….

So remembered (and called myself various names) that we have a Threat Management Gateway that presents OWA to the internet.

So after a quick search on the TMG and a few clicks of the Exchange OWA web listener I found:

Swapped the gif file and BAM, still didn’t work!! ARRRRGGGHHH

Here’s the clincher, I had to restart the TMG service!! Unfortunately TMG caches these pages and restarting the service is the only way to clear the cache as far as I know.

So BAM it works!!  🙂

Tags: , ,

Excess Exchange 2010 activity

Posted by robd on February 15, 2013
Apple, exchange 2010 / No Comments

So Apple have released a fix for the recent issue with over logging issue on Exchange 2010.

Here’s the link: http://support.apple.com/kb/TS4532

But for the lazy here’s their fix:

  1. Go to Settings > Mail, Contacts, Calendars
  2. Select the Exchange account from your Accounts list.
  3. Turn the switch for Calendars to OFF.
  4. Wait ten seconds.
  5. Turn the switch for Calendars back to ON.

Tags: , , ,

Exchange 2010 – Health Report

Posted by robd on February 14, 2013
exchange 2010, powershell / No Comments

So the other day I wanted to run a health report on Exchange servers to show management that our Exchange environment is happily chugging away and their email problems are likely be down to inferior laptops or iPads or what ever!

Tha’ts when I found a brilliant PowerShell script written by Paul Cunningham that runs a health report on all the Exchange servers in your environment and emails you a report.

You’ll need to setup the email bit in the script, for example:

SMTP
Then just run it, for example:

The it produces a very useful report:
report

 

Tags: , , , ,

Exchange 2010 – transaction log files & iOS 6.1

Posted by robd on February 11, 2013
Apple, exchange, exchange 2010, iOS / No Comments

So anyone who looks after Exchange 2010 and allows users to connect their iPhones and or iPads will know the fun and games of mail going missing, calendars loosing invites or not matching their outlook calendars (most the above can be fixed by recreating the users mail profile on the ipad or in Outlook)….

Well to top this off iOS 6.1 seems to cause a sync loop on the Exchange mailbox server causing excessive transaction logs (thousands of logs in 15 minutes) which can lead to a very dead exchange server.

To combat this you’ll need to find the device and then turn off active sync for that user, or another route would be to block all iOS 6.1 devices from establishing a connection.  Edit: some people say removing the exchange mail profile from the iOS device and re-adding it solves the issue!!

To find the users, I’d first run a report of who has and hasn’t updated their devices by running a script that I’ve previous mentioned:

http://bohemiangrove.co.uk/exchange-2010-with-apple-ios-6/

Output:

All_iOS_Devices_on_Exchange_2010

Alternatively if you’re scared of PowerShell or maybe a admin has blocked you for using it because you’ve been a naughty boy or girl you could use Log Parser Studio which is like a plugin for Log Parser!!  So install Log Parser on a CAS server and run Log Parser Studio, then point the tool at the IIS logs.  Finally use one of the built in search tools such as ActiveSync Report [Top 20] to find the which device has produced large amounts of hits and hence caused all the transaction logs!

Log_Parser_Studio

So know you have a list of users who have iOS 6.1 you could disable active sync for each user via the Exchange control panel.

Or if you’d rather block all the iOS 6.1 and any future 6.1 devices you could run this the following PowerShell command that will create an Active Sync rule.

Note, if you don’t warn users they may complain about their devices not accepting passwords or similar!!

Tags: , , , , , ,

Exchange 2010 with Apple iOS 6

Posted by robd on January 15, 2013
exchange 2010 / 1 Comment

Since Apple update iOS to version 6 earlier this year I’ve had nothing but problems with calendar being mismatched between Apple devices and Outlook and users claiming mail has completely disappeared off the planet!!

Urggghh is my initial answer…..Now I try and tell users I really don’t think its Exchange as we patch like crazy and other devices such as Android and Microsoft work like a charm… But some fan boys don’t care what I have to say and blame me or MS anyhow!

The truth as far as I know it is its down to Apples adaptation of Active Sync that as far as I can tell must work totally different and seems to have been a bit of a after thought to the Apple product line.  Just for a info a quick and temporary solution is to remove the Exchange account from the iOS device and re-add it!

Interestingly MS don’t want ownership of these issues:

http://blogs.technet.com/b/exchange/archive/2012/10/23/ios6-devices-erroneously-take-ownership-of-meetings.aspx

Quote:

Tell users not to take action on calendars on iOS We’re not seeing this particular issue if users don’t take action on their calendar items (for example, accept, delete or change meetings).

Block iOS 6 devices Exchange server comes with the Allow/Block/Quarantine functionality that enables admins to block any device or user.

Also here’s a list of some issues:

http://support.microsoft.com/kb/2563324

Anyhow, to try and get a hold on the situation and find out all the iOS devices connecting to the Exchange environment (so I know whos going to moan at me at some point) I found a brilliant power-shell script by Steve Goodman, here:

http://gallery.technet.microsoft.com/Exporting-iOS-6-and-3d4ac87b

 

The Script:

Tags: , , , ,

When Exchange 2010 Breaks

Posted by robd on January 11, 2013
ESX, exchange 2010, powershell, WAN / No Comments

Part 1

So recently a client had some WAN issues where the WAN kept dropping packets on the primary link, meaning some network traffic would work fine and other network traffic would fall off the face of the planet!!

 

Unfortunately the system affected the most was Exchange 2010!!  The site had Exchange 2010 DAG set-up between two sites.  One on Site A and another on another site B with the File share witness on Site A.  So because some traffic was flowing nicely and some wasn’t some of the cluster services were up and some went down meaning the DAG got in a pickle!

 

Site B ended up taking ownership of the DAG and wouldn’t let go because he file share witness wasn’t available.  So as people could still get mail, I patiently waited for the WAN to be fixed (in the end someone pulled the primary link which forced the secondary to take over and everything became hunky dory).

 

So now the WAN was back and happy but still the cluster service wouldn’t appear, so after digging the arp entries were missing (no idea why), so ended up creating another IP for the cluster service….Done.  It’s up, time to reseed!

 

Update: Well shit me, MS know about this:

http://support.microsoft.com/kb/2743761

 

Well anyway I wish the reseed was that easy! During the issues, the mail store was un-mounted and now it won’t mount!! Booo, so another admin decided to run eseutil and recover the DB which worked but then decided to seed the wrong way meaning site B now has an old mail store with no mail from that day!!  BUGGER!

 

Part 2

 

So I need to rerun all the mail and then import and luckily all the missing mail logs are still on Site B!

I know what you’re thinking:

 

Restore the EDM file from a backup,

Rerun the Logs via ESEUTIL,

Mount the recovery DB,

Merge users,

Bam.

 

Well the business was very panicky now and didn’t want anyone touching anything on Exchange till things settled down, fair enough a day’s worth of mail is missing at the moment so people are touchy.

 

So I decide to make use of the most excellent Veeam feature SureBackup, which will allow me to restore the entire exchange live setup including a DC in lab environment without touching the live environment!!  Brilliant!

 

So I set it up, add a host, some storage, make sure I have a good backup of Exchange and it runs, great!!  Be careful though as it uses snap shots heavily so every change adds to a delta file meaning storage gets eaten!!  So here I am with the live environment in a lab environment, there some issues as I cant restore site B, so I have to remove the DAG and the other site which I do via ADSI edit (quick and easy but dangerous)!

 

So I copy over the recovered DB I restored with file level restore on veeam (copied by creating a HD in ESX and mounting it on veeam and copying file to it then mounting in my lab),

Create a recovery DB:

 

Checked the logs:

 

Replayed:

 

(May of run ESEUTIL /p too, cant remember)

 

Mounted!

 

Emails are all there but in the disconnected items section, reconnect to a test db and GREAT the emails are there!!

 

So connect all the mailboxes (manually unfortunately), then export them to pst (to my mounted HD):

 

Un-mounted the HD, added the HD to the live environment and imported the PSTS:

Tags: , , , , ,

Copy Protected by Chetan's WP-Copyprotect.