David M. Stein's Blog

Windows Server 2008 Reliability: Put up or Shut up

David M. Stein — Thu, 28 Aug 2008 01:35:00 GMT

Ok, so I was thinking about my anecdotal ranting in my previous posting (about Windows Server 2008 reliability). Stupid me. I forgot about the nifty-cool "Reliability and Performance Monitoring" feature in WS08 (and in Vista also). If you haven't looked at that: YOU SHOULD. Microsoft really took a risk with this feature. Talk about unzipping your fly and letting it all out for everyone to see. This utility captures EVERY failure or impediment to your system's stability or reliability. Drivers, applications, updates, hardware, and of course Windows itself. It scores from 0 to 10 on overall reliability. Every time a patch is installed, it hits the score. Every time you lose facility power, it hits the score. Nothing is spared. Nothing is excluded.

Here are some screen shots of my WS08 box at home. Note that this box is meager in terms of hardware. It's an old Dell OptiPlex GX-260 with 1 GB of RAM, a 40 GB IDE drive for C: and a 500 GB IDE drive for data. It is configured as my AD domain controller, and hosts IIS7 for internal web apps, and also WSUS 3.0 SP1 for managing all the clients on my network (there are four). The data drive stores and serves up our family documents, photos, music and so forth. I back it up fairly often, but someday when I have enough money to spare I would like to get a better rig. For now, food and mortgage come first (and Sallie Mae insists on being second).

Figure 1 below - From back in March 2008, when I first took the time to establish the baseline. Note the near-perfect score during that period. The term "Disruptive shutdown" refers to power outages affecting our entire house. I have a CyberPower UPS but didn't install the correct driver for it on WS08, so it really didn't help do an automatic shutdown. It just beeped until I got fed up and went over to do a shutdown. However, as you might guess, "disruptive shutdown" means I didn't bother and just let the UPS die and take the server down with it. Not ideal, but that's not unusual in my house either.

Figure 2 below shows late June to early July 2008, and another "disruptive shutdown" and you can see a string of eight "Information" icons in the Software (Un) Installs row. Those were Windows updates being installed (successfully). The two "application failures" were from a trial version of an unnamed application which was later removed. Those definitely hurt my reliability score as you can see. Keep in mind that even after correcting a problem it takes a while for your score to recover. This makes sense actually and it's a good thing since it keeps the scoring as realistic as possible.

Figure 3 below is another screenshot from early August 2008 showing the impact of installing Silverlight updates. It's interesting to see how my reliability score is still gradually recovering from the mess you see in Figure 2 above.

Figure 4 shows my current reliability score of 8.68 as of today (August 27, 2008). The yellow warnings are from me removing remnants from the unnamed application that caused all the issues in Figure 2.

Maybe you're not impressed, but I am. Not only because of the scores, given the crappy hardware this is running on and all the roles which are loaded, but I'm also impressed with the Reliability and Performance Monitoring features. The only thing I would want to see added in the future would be a way to print or export the reports to something like XML, XHTML, PDF, XPS or even Excel. Right now you can't export anything and there's no Print option either. Print-Screen is about your only built-in option. Regardless, this is cool stuff.

Share this post:

WHS or WS08: Depends on Your Quirks

David M. Stein — Wed, 27 Aug 2008 19:26:00 GMT

There's a lot of cool stuff going with Windows Home Server (WHS) lately. Especially now that PP1 has been out for a while and added some fixes and improvements. This short blog post by Paul Thurrott is but one example of the coolness WHS provides. But what caught my attention more than the backup and restore feature, was the comment about it sitting on a shelf. The implication being that you don't need to babysit, or even really tinker with it. It's become what I've always said a home computer needs to become in order to gain mass appeal: an appliance. Yes. Like a thermostat. You occassionally adjust it, but for the most part it does its thing and you don't have to worry about it. WHS is aimed in that direction obviously.

But in some respects so is Windows Server 2008. When the beta program ended, I did the celebratory upgrade to the RTM version and continued on. Happy. Content. Aside from just poking into the inerds to see what they look like, and how they work, I rarely have to tinker with it. It just runs. And to be totally honest here, aside from power outages (storms, etc.) my cheapy little workhorse WS08 server has NEVER had to be forcibly restarted. It has NEVER died. NEVER locked up either. Sure, patches may require a reboot here and there. But I don't have to peak through my fingers covering my eyes to worry about what scarry, obscure error and warning events are lurking in the event logs. I just don't see them very often. I don't have to restart hung or failed services. I don't have to force a restart. In short - It just friggin RUNS.

One test I did was to run a Suse 10 and a Ubuntu 8 server for several weeks without touching them and see whether WS08 required any more restarts. After four weeks, all three servers were prompting me for updates and a restart. But none had restarted for any reason on their own. Is this de facto proof that WS08 is "as reliable" as Ubuntu or Suse? No. But it did prove to me that unlike WS2K and WS03 (which is quite a bit improved over WS2K), WS08 was much more stable and worry-free. And it has remained so for as long as I've been using it. Keep in mind that this "workhorse" server of mine is a rather beat-up Dell OptiPlex GX 260 with 1GB of RAM. It serves up my photo library, music and documents and pretty much everything. God, I love it. Just thinking about it, I drift off into a foggy dream state. Me and my WS08 box running in slow-motion towards each other in a sunny field of tall grass. Someone hand me a tissue please?

Yes, I know, this went totally stupid at the end. Oh well.

Share this post:

Windows Server 2008 Virtual Appliance for VMware

David M. Stein — Tue, 26 Aug 2008 12:14:00 GMT

Have you heard someone around you say something like "I would really like to play with Windows Server 2008, but I can't afford it"? Or maybe "I would like to play with WS08 but I don't have a *server* laying around to try it on." ?

You can hold their hand, offer a tissue and tell them there's help. The VMware Virtual Appliance Center has plenty of useful things to offer. One of them is a pre-built virtual machine with Windows Server 20008 and VMware Tools installed. 100 percent legal, evaluation version. And this is the "Enterprise" edition. So, you can download Virtual Server 1.x (or Virtual Server 2.0 RC) for FREE, and download a ready-to-run WS08 virtual appliance for FREE and start getting up to speed now. What are you waiting for?!

Share this post:

WSUS Role Available for Windows Server 2008

David M. Stein — Tue, 26 Aug 2008 10:45:00 GMT

I think I blogged about this way back when WS08 was being released, but here it is "again": A download (32-bit, or 64-bit) which adds the role Windows Server Update Services (WSUS) 3.0 SP1. This was actually something that was available during the final stages of the WS08 Beta program. I was so eager for them to make this part of the initial RTM but it didn't happen and there wasn't any reason given for that at the time. I figured it was due to either something not being quite ready for RTM disk pressing or a bug was found. Who knows. In any case, this is nice to have even if it wasn't in the initial media release. Don't bother clicking on the "Knowledge Base Article" links (to 940518) because it hasn't been posted yet (as of Aug 26, 2008 anyway).

So, what's the big deal? Well, the "big" in "big deal" is entirely subjective, right? Of course it is. To some, it's a big deal because it allows you to add the WSUS server capability by checking a box in the Server Roles menu. For others it's a sleep inducer. I think it's somewhere in between. Whatever, if you have time to read this, you obviously don't have enough to do and should probably go ahead and download it and check it out.

Share this post:

WSUS Itanium Cranium 64-bit Drainium Insanium

David M. Stein — Sun, 24 Aug 2008 15:49:00 GMT

Holy crap! This is getting tedious. I do not have any 64-bit or Itanium machines in my environment. If I did, it would signify one of two things: (A) I would have WAY too much time to play with computers and no life outside of that, or (B) my family packed up and left me alone in a prison on an island with nothing to do but work on 64-bit or Itanium computers.

WSUS 3.0 has indeed been a huge improvement over SUS 1.x and WUS 2.0, but there are still shortcomings you have to deal with. One of these is that it is incapable of excluding 64-bit and Itanium patches. I don't care what you do. You can select products and categories around those platforms all you want, they will still show up. It's sort of the Night of The Living Dead, part 20. Remember when SUS and WUS would choke on all the languages of .NET framework updates? You would be grinning away like you won the lottery whilst selecting only English in your options setup, only to drop your jaw on the floor in complete dismay over Korean, Croatian, Serbian and Japanese versions of the patch showing up everywhere. The fun of it all. Well, they didn't want you to feel left out in WSUS 3.0 so they've brought back ill-filtered updates for your enjoyment. Granted, it's not totally WSUS's fault. A lot of the blame rests on how the patches are built and cataloged. Many just do not publish their properties in a granular manner to allow for such filtering. That was the case with the .NET framework updates.

Here's the fun. If you're like me and don't manage any 64-bit or Itanium computers in your environment, you have to use the semi-functional "Search" feature and look for strings "x64", "64-bit" and "Itanium" and then select-all and right-click and choose Decline. I sure hope they do something to make it easier to avoid synchronizing 64-bit and Itanium updates upfront so I don't have to keep declining them all them time to keep my lists clean.

Share this post:

SCUP: Publish Your Own Patches to Your Own Applications

David M. Stein — Sat, 23 Aug 2008 02:34:00 GMT

System Center Updates Publisher is what SCUP stands for. Once again, Microsoft didn't fund the Acronymn Department sufficiently this year. Remember when SUS almost became WUS? Thankfully they moved on to WSUS at least.

In their own words:

SCUP is a stand-alone tool that enables independent software vendors or line-of-business application developers to import software update catalogs, create and modify software update definitions, export update definitions to catalogs, and publish software updates information to a configured update server. By using updates Publisher to define software updates and publish them to the update server, administrators can begin detecting and deploying published updates to client and server computers in their organization.

Download it FREE at the Microsoft Download Center.

So all you in-house corporate developers out there, stop your whining about getting Microsoft to back you up with a tool to help you deploy your own patches to your own crap, oops, I mean software. I only have one question: What the heck is a "detectoid"? Is that like a bad medical condition, or is a new flavor of Altoids?

Read the online documentation

System Center Updates Publisher Catalogs

Share this post:

Sssssssssss...... Boom! Windows 7 Lands on WSUS

David M. Stein — Fri, 22 Aug 2008 00:30:00 GMT

That's right. "Windows 7 Client" is now a new category within WSUS 3.0 consoles everywhere (well, as soon as they get their next synch). The heads-up courtesy of the WSUS Team Blog. Ok Ok, I'm overreacting a bit here I know. It's not really "here" yet, but we can all drink a beer to celebrate, right? As if we need a reason.

Share this post:

System Center Configuration Manager 2007 - Sample Configurations & Common Performance Questions

David M. Stein — Thu, 21 Aug 2008 01:12:00 GMT

Phew! Long title for sure. Here's a stupid challenge for you: Drink three imported beers in quick succession and repeat the title of this post ten times as fast as you can. Go ahead, I'll wait...

I ran across this on the TechNet blog site and decided to stop and read through it. This is a rather handy little document actually. It breaks down a few sample scenario-type SCCM setups and provides some basis for comparison as far as infrastructure, servers and objects, etc. What's nice is that they actually outlined a profile for a single-server site with 10,000 clients. That's the first time I've ever seen anyone officially state that this was something not laughable. Not that SCCM can't handle 10k clients from a single site server, but it seems (to my puny little world of experience, ok, stop laughing...) that anytime some engineer heard "10,000 clients" and "SMS" in the same paragraph, it immediately was translated into multiple sites or at least multiple site servers.

I once deployed a single SMS 2003 site with one primary/central server for just under 3,000 clients, spread over 22 continental US locations on pretty decent WAN links and was scoffed at by some. That environment used existing local site file servers as DP's so it was able to deploy things without choking the WAN. However, I got a boost of confidence two (unnamed) sources inside of Microsoft actually gave my plan the nod of approval. That was 3,000. This document starts at 10,000. The top scenario is a multi-site environment with 250,000 clients.

Only 18 pages to read and well worth the time. Download the document here.

Share this post:

Windows Server 2008 Core Survival Kit

David M. Stein — Mon, 18 Aug 2008 22:20:00 GMT

Ok, not really a "survival kit" in the traditional sense. I don't have a magic download widget like CoreConfigurator or anything. This is actually more like an emergency list. You know, like if you live in areas where you need to keep a list of things handy in case the "big one" happens and you risk losing your house/apartment, etc. Like a hurricane or typhoon or raging fire.

Mark Russinovich once described WS08-Core as being more of a surgical modification than a ground-up design. Look for a more robust and architected "core" platform in the next Windows Server release. The more time you spend poking around in WS08-Core the more this will become obvious to you. However, it is still an impressive offering and while Linux/UNIX fanboys may scoff at it, it's hard to argue against how significant this is for Microsoft to have finally made a 180-degree turnaround and actually make good on such a promise. Indeed, the benefits are real: smaller overhead footprint, better performance, and increased security resulting from a smaller attack surface.

So if you plan on using Windows Server 2008 Core, or as I refer to it "WS08-Core", you need to become familiar with the following features and aspects:

SLMGR.vbs - A utility script for resetting a license, extending the eval period, and activating a WS08-Core license
CSCRIPT.exe - The age-old script engine. You will use it quite often to do chores you find painful to do otherwise.
CONTROL.exe - Used for invoking select (traditional) Control Panel applets such as time and date, regional settings, etc.
OCLIST.exe - List the roles and features installed on the server
OCSETUP.exe - Add or Remove roles and features on the server (you still need DCPROMO for making a domain controller)
PKGMGR.exe - Vista and WS08 Package Manager is a general maintenance utility (refer to TechNet for more)
LOGMAN.exe - The CLI version of Performance Monitor (see TechNet for more)
ServerManagerCMD.exe - A CLI version of the WS08 Server Manager, sort of.
NETSH.exe - Traditional network configuration shell
DCPROMO.exe - Promote and demote a domain controller (role must be enabled using OCSETUP first)
SC.exe - Create and manage services (refer to KB251192 for more)
SCREGEDIT.wsf - A general utility script for managing registry settings (check out Guy Teverovsky's blog for some good info)
NETDOM.exe - A general utility for managing domain computer
WEVTUTIL.exe - The CLI version of Event Viewer (refer to TechNet for more)
WBADMIN.exe - The CLI version of NTBACKUP (refer to TechNet for more)

An example of using ServerManagerCMD would be to add the PowerShell scripting feature to your WS08-Core server. The syntax would be:

#> servermanagercmd -Install PowerShell

Note that while Server Core is touted to be a CLI-based operating system, it's really more "Core-ISH" than true "Core". There are still some features which have not be ported to a CLI paradigm:

Registry Editor (regedit.exe)
Notepad
Task Manager (taskmgr.exe)
Time and Date
Regional Settings

There are many more commands you can learn and employ to your benefit. I can't overstate the importance of the infamous command utilities like IPCONFIG, NSLOOKUP, REGSVR32, SHUTDOWN, TASKLIST or ICACLS.

Keep in mind that some commands listed work on WS08 (GUI) only, such as WINVER, while others work on both or only on WS08-Core. Some common tasks are handled differently as well. For example, you can't just jump into DCPROMO and make a domain controller with WS08-Core. As with WS08 (GUI), you must first add the Active Directory Domain Services role to the server, then use DCPROMO to promote it to the domain controller "role". However, with WS08-Core, there's no "wizard" to walk through when using DCPROMO. You need to first prepare an "answer file" with all the desired parameters spelled out, and then you hand it to DCPROMO to do the processing. This is only a small sampling obviously, and you will find much more information as you work with WS08-Core yourself. Hopefully this gives you enough to get going on your own for now. For more tips and information check out this article on TechNet.

It's often easier to manage WS08-Core from another WS08 (GUI) server or Vista client using remote administration tools, but knowing how to effectively administer WS08-Core at the console is definitely something to know.

Share this post:

Setting Automatic Updates for Windows Server Core 2008

David M. Stein — Sat, 16 Aug 2008 00:55:00 GMT

There are quite a few source on the Internet describing how to turn on Automatic Updates inside Server Core. Most cover using the SCREGEDIT.WSF script to invoke the /AU option and set the general Automatic Updates option (1 to 4).

cscript c:\windows\system32\scregedit.wsf /au 4
net stop wuauserv
net start wuauserv

The value of 4 indicates automatic check for updates, and download and install at scheduled day and time. Then run wuauclt /detectnow to initiate a local scan and validation.

However, if patches are found to be needed, you won't be prompted to install them. That's one of the idiosyncracies of working in a command shell environment.

It would be nice if they beefed wuauclt up a bit for this environment, maybe to actually display some useful information (status, results, etc.). Here's my suggestions (as if anyone at Microsoft reads my gibberish)...

- wuauclt /detectnow
- automatic updates source: https://update.microsoft.com
- scanning for updates... (0% complete)
- updates found for this computer: 4 critical, 6 security, 3 device drivers, 12 optional
- install updates now? /N: _
- installing updates... (0% complete)
- updates have been installed.
- computer restart is required to complete updates... restart now? /N: _

I may be smoking crack of course, but I can dream, can't I? But what if you don't want to keep the default settings for which days or times to do a scan/install? The defaults are stored in the registry, in the same places found on a "normal" Windows Server 2008 machine. However, as with any Windows computer, this doesn't apply if the computer is being configured via Group Policy.

To change the day and time settings, simply enter "regedit" in the command console to open the Registry Editor. Like several other commands, this is still a GUI feature in the so-called GUI-less product (along with Notepad and TaskMgr). Navigate to the following location to...

HKEY_LOCAL_MACHINE \ SOFTWARE \ Microsoft \ Windows \ CurrentVersion \ WindowsUpdate \ Auto Update

The settings are ScheduledInstallDay and ScheduledInstallTime and use the same values as any other Windows client.

Share this post:

Installing VMware Tools inside a WS08 Core guest

David M. Stein — Sat, 16 Aug 2008 00:12:00 GMT

So, I couldn't resist this post on Daniel Petri's web site describing how to install VMware Tools "additions" (chuckle) inside a Windows Server 2008 Core Edition guest machine. The thing that caught my attention was the caveate notes about having to hack it (a little)...

Note: Some websites claim that the installation of VMware Tools will stall, and that in order to complete it successfully you need to manually kill the RUNDLL process in Task Manager. Other sites claim that you need to run the installation by typing the following command:

Don't take this as criticism of the article, it's very good indeed. Actually, the problem is that the installer stops to prompt for three issues and is waiting for the user to click "ok" or "yes" or whatever. However, the z-order of the popup alerts puts them behind the installer dialog, so you never see then unless you just suddenly think to move it around in case it's hiding something behind it (it is!). The first two prompts involve DLL registration errors. The third asks if you want to update the Microsoft HTML help engine to display VMware Tools help content. Once you click past the two errors (they're harmless actually, both involve GUI features, so hey) and say "No" to the HTML Help update request, you're off to the finish line and the usual reboot prompt.

Share this post:

WSUS 3.0 API Samples Download

David M. Stein — Wed, 13 Aug 2008 20:24:00 GMT

Cool-er-est. Gnarly. Bomb-diggity. Microsoft posted a neat-o set of API sample/examples for WSUS 3.0 on the Download Center yesterday. Among the goodies packed inside this gem are those listed below. I like the CleanStaleComputers feature. Too bad they don't yet have a CleanStaleJokes feature.

Sample Name	Language	Description
ADImporter	Visual Basic	This tool allows you to pre-populate target groups with computers from Active Directory. The sample application connects to the default Active Directory domain controller, and lists the OUs and computers in the domain. It also lists the computer groups available on the WSUS server. Using this sample, you can select computers from Active directory and "register" the selected computers with Update Services. Note that each computer registered must be configured to talk to the WSUS server independently of using this tool.
ApprovedUpdatesToXML	Visual Basic	Creates an XML file with a list of approved updates and details of their approval including the groups that the update is approved for, and the approval action.
ApproveForOptionalInstall	Visual Basic	Approves a specified update for "optional install". This causes the update to be registered in the "Add/Remove Programs" Control Panel applet (also known as "Programs And Features" in Windows Vista) of the client machine. The user can choose to install the update using the Add/Remove Programs Control Panel applet.
CleanStaleComputers	Visual Basic	This sample application removes computers from the WSUS server that have not contacted the server in a specified number of days. USAGE: CleanStaleComputers /DAYS:[1-365] /DELETE:{YES \| NO} /PROMPT:{YES \| NO} /DAYS. Days since the computer contacted the server /DELETE. Delete from the WSUS server or move to the Stale computers group /PROMPT. Prompt before moving/deleting computers
ComputerStatusToXML	Visual Basic	Creates an XML file with a list of computers on the WSUS server, and the status of the updates approved for each computer. The list can optionally include computers that are rolled up from downstream WSUS servers. USAGE: ComputerStatusToXML.exe [IncludeDownstreamComputers]
ComputersNeedingReboot	Visual Basic	Creates an HTML report that lists computers that have installed updates, but still require a reboot to be fully patched
ListApprovedUpdates	Visual Basic	Displays a list of updates that have been approved after a specified date and also creates a text report that lists the updates.
UpdateStatusToCSV	Visual Basic	Creates a simple CSV file with a list of approved updates and a summary of the status of the updates on each computer
UpdateStatusToXML	Visual Basic	Creates an XML file with a list of updates, with update status for each computer client.
UpdatesToXML	Visual Basic	Creates an XML file with a list of updates and information about each update, including ID, title, description, KB articles etc.
WsusMigrate	Visual C#	WSUS Migration Tools can assist in migrating update approvals and target groups from one WSUS server to another. See the ReadMe under the WsusMigrate folder for more information on these tools.

Share this post:

WSUS-ers: Get Ready, Get Set: Patch Time

David M. Stein — Sat, 09 Aug 2008 01:10:00 GMT

Microsoft will once again be unleashing a pack of updates on the next super "Patch Tuesday" on August 12. Details are posted here. So, be prepared brave patchmeisters. And drink heartily beforehand. Angry users and crashed computers don't hurt as bad when you're sufficiently intoxicated. As always, Uncle Dave says "Remember Kids: Test those patches before you drink"

On a somewhat related sidenote: I used to assume that the whole world was familiar and cozy with using virtualization in their environments. I mention this because virtualization goes with patch testing like ham and eggs, or grenades and kevlar, or guns and liquor, or well, whatever, you get the picture. However, upon coming aboard my current employer it has become painfully obvious that this is not true. Much of the technology world (indeed, many places you would expect to be way ahead of the curve) are actually still catching up. Which flavor of virtualization you prefer doesn't matter. It really doesn't. It's more important that you just get started and make it an integral part of your IT world. Patch testing is usually step 1. That's a natural place for virtualization to be utilized. Next would be application testing and package deployment testing. This is great for traditional packaging toys like Wise or InstallShield, as well as new-fangled wunderkind contraptions like SoftGrid (aka Microsoft App-V), ThinApp, XenApp, SVS or whatever.

Oh boy have I slipped off the wagon and gone off topic. I apologize. I do that quite a bit when I get excited.

Back to the topic: Security Patches! ...

SO, what's in store this time around? Seven (7), count 'em, "critical" patches, each offering to plug a "remote code execution" vulnerability in a variety of things. You should seriously get these tested and rolled out. There are also five (5) "important" updates which shouldn't be ignored. The platform applicability list covers XP, Vista, WS03, and so on. Note that only two "critical" updates apply to Vista and Vista SP1 clients. I hope that someday soon all the FUD about Vista will vanish. It's really a cool product, but Windows Server 2008 is cooler.

Share this post:

Level 5 IT Guides Open Access for Limited Time

David M. Stein — Fri, 08 Aug 2008 23:38:00 GMT

You may have heard this already, but on August 13, 2008, the Level 5 IT Guides will be opened up to allow everyone to view sessions which are normally reserved for paying subscribers. This will be done in two two-hour sessions, one in the morning and one in the evening (US central daylight time). Check out the Details Here.

If you haven't seen the Level 5 IT Guides this will be a great opportunity to do so. Feel free to post questions or requests for additional sessions. I have a bunch of sessions on WSUS 3.0 and Windows Server 2008, but there's obviously room to add many many more. But I need input from you to help narrow my list to producing content that matters to you. The same is true for the other authors I'm sure.

Share this post:

My WSUS wishlist

David M. Stein — Wed, 30 Jul 2008 01:58:00 GMT

Some things I'd like to see added to a future version of WSUS. I've posted these on the WSUS.info forums also.

Enhanced filtering views. Right now, to find all the updates which contain "ia64" in their title, you have to use the search feature. While it is helpful, it's also tedious at times. The ability to pin a filtered view for arbitrary wildcard string matched results on the Updates collection would be very handy.
A DNS/NetBIOS name resolver for the "upstream server" entry. Next to the edit box where you normally type in the name of an upstream (parent) server, it would be nice to verify that WSUS can in fact resolve the name correctly. While it's easy to do with a CMD console opened, having a "Resolve" button next to the box would also be handy.
A built-in Backup/Restore Wizard. Yes indeed. This could be very handy, especially when you need to replace a WSUS server. Something to bundle up the database as well as all the configuration settings.
The ability to filter out or exclude ia64 and x64 updates. Right now you simply cannot do this. You have to list them using the Search feature and then highlight them and choose Decline (at least in my case). Hence the first bullet item.

Share this post: